Deep Low-Rank Coding for Transfer Learning

1 Introduction

In machine learning and pattern recognition fields, there is always a situation that we have plenty of unlabeled data while no or insufficient labeled data for training in the target domain. Transfer learning [Pan and Yang, 2010] has been demonstrated as a promising technique to address such difficulty by borrowing knowledge from other well-learned source domains, which might lie in different distributions with the target one. Many recent researches on transfer learning have witnessed appealing performance by seeking a common feature space where knowledge from source can be transferred to assist the recognition task of target domain [Chen et al., 2012; Ding et al., 2014; Shao et al., 2012; Shekhar et al., 2013; Long et al., 2014b]. Therefore, it is the key to uncover the rich and discriminative information across source and target domains in transfer learning.

∗This research is supported in part by the NSF CNS award 1314484, ONR award N00014-12-1-1028, ONR Young Investigator Award N00014-14-1-0484, and U.S. Army Research Office Young Investigator Award W911NF-14-1-0218.

Recently, low-rank constraint [Liu et al., 2013] has been widely studied in conventional transfer learning due to its locality aware reconstruction property, meaning that only appropriate knowledge is transferred from one local space in the source/target to another local space in the target/source. Two representative methods are LTSL [Shao et al., 2014] and L2TSL [Ding et al., 2014], which explicitly impose lowrank constraint on the data reconstruction or latent factor in a learned common subspace. Those methods only employ a shallow architecture containing a single layer. However, knowledge transfer can be better learned frommultiple layers with a deep structure.

Most recent researches on deep structure learning to capture a better feature representation attract increasing interest [Chen et al., 2012; Nguyen et al., 2013; Zhou et al., 2014; Chen et al., 2014], since discriminative information can be embedded in multiple levels of the features hierarchy. In fact, this is one of the major motivations to develop deep structure learning framework, so that more complex abstraction can be captured. However, current deep transfer learning methods failed to align different domains and learn deep structure features simultaneously. Without any knowledge about target domain, the feature extraction process performed on the source data would definitely ignore information important to the target domain.

In this paper, we propose a Deep Low-Rank Coding framework (DLRC) for transfer learning. The core idea of DLRC is to jointly learn a deep structure of feature representation and transfer knowledge via an iterative structured low-rank constraint, which aims to deal with the mismatch between source and target domains layer by layer (Figure 1). Our main contributions are summarized as:

• A deep structure is designed to capture the rich information across source and target domains. Specifically, the deep structure is stacked by multiple layer-wise lowrank codings. Therefore, it can refine features for source and target in a layer-wise fashion and preserve more essential information to the target domain.
• An iterative structure term is developed for each Singlelayer Low-Rank Coding (SLRC), which works in a local-aware reconstruction manner. Through labeling

most confident samples in target domain, the learned features become more discriminative, since the marginal and conditional disparities are both leveraged.

• Marginal denoising regularizer is incorporated to guide the low-rank coding by seeking a robust and discriminative transformation shared by two domains, which is jointly optimized with low-rank reconstruction by uncovering rich information from complex data across two domains.

2 RelatedWork

In this section, we briefly discuss some related works, and highlight the differences between them and our method.

Transfer learning has been widely discussed recently and for the survey of state-of-the-art methods, please refer to [Pan and Yang, 2010]. Recently, low-rank transfer learning has been well-studied to ensure that accurate data alignment is achieved after data adaptation [Shao et al., 2014; Ding et al., 2014; Ding and Fu, 2014]. The low-rank constraint enforced on the reconstruction coefficients matrix between domains is able to reveal underlying data structure, especially when the data lie in multiple subspaces, which can guide the conventional transfer subspace learning. Different from existing methods in this line, we introduce iterative structure learning to recover the low-rank structure of the coefficient matrix in a supervised way. Furthermore, we employ the low-rank constraint on the data transformed by a mapping learned from marginal denoising regularizer, and therefore our method is more robust to corrupted data.

Most recently, the thought of deep structure is incorporated into transfer learning to uncover the rich information across domains. Chen et al. developed marginalized Stacked denoising Autoencoder (mSDA) to learn a better representation by reconstruction, recovering original features from data that are artificially corrupted with noise [Chen et al., 2012]. Zhou et al. managed to learn a feature mapping between cross-domain heterogeneous features as well as a better feature representation for mapped data to reduce the bias issue caused by the cross-domain correspondences [Zhou et al., 2014]. In this paper, we also adopt the thought of deep transfer learning, however, our method jointly learns the low-rank codings and transfers knowledge from source to target in a unified deep structure framework. By stacking multiple layers' low-rank coding, we build a deep structure to capture more discriminative features across two domains.

3 Deep Low-Rank Coding

In this section, we first briefly discuss our motivation, then propose our single-layer low-rank coding with its solution. Finally, we introduce our deep low-rank coding framework by stacking single-layer low-rank coding to multiple layers.

3.1 Motivation

Recently, mSDA [Chen et al., 2012] and its variants [Zhou et al., 2014], achieve exciting recognition results for transfer learning by extracting layer-wise features across different domains. These works stack marginalized denoising Autoencoder (mDA) layer by layer to capture the rich and discriminative features. mDA has shown the effectiveness in transfer learning and proven to be much more efficient [Chen et al., 2012], due to its linear property.

Considering previous work only learn deep structure feature [Chen et al., 2012], or separately learn feature and transfer knowledge [Zhou et al., 2014], we propose to refine layerwise features and align different domains in a unified framework. In such way, knowledge from source domain can be transferred to the target one layer by layer, which guides lowrank coding to produce more discriminative and important feature to the target domain. In the following sections, we will present our Deep Low-Rank Coding (DLRC) based on Single-layer Low-Rank Coding (SLRC).

3.2 Single-layer Low-Rank Coding

Given a set of target domain XT = {xT ,1, · · · , xT ,nT } with nT unlabeled data points and a set of source domain {XS, YS} = {(xS,1, yS,1), · · · , (xS,nS , yS,nS )} with nS labeled data points and YS is the label vector. Assume X = [XS,XT ] ∈ Rd×n , where d is the original dimension of two domains and n = nS +nT is the total size of two domains.

Our Single-layer Low-Rank Coding (SLRC) adopts the thought of conventional low-rank transfer learning [Shao et al., 2014; Ding et al., 2014] to seek discriminative lowrank codings. With its locality-aware reconstruction property, marginal distribution divergence across source and target domains would be reduced so that well-established source knowledge can be passed to target domain. Therefore, we develop the following objective function as:

where rank(Z) is the operator to calculate the rank of lowrank coding matrix Z ∈ RnS×n , which can be solved with nuclear norm [Liu et al., 2013]. W ∈ Rd×d is the transformation matrix (or rotation) on original data shared by two domains. Ω(W) is the loss function concerned W and λ is the trade-off parameter.

To seek a better transformation matrix W in low-rank constraint, we incorporate recent popular mDA [Chen et al., 2012], which is designed to seek a mapping W from the original data to the corrupted one so that the learned W is robust to corrupted data. mDA has an advantage on efficient performance and small computational cost, whose objective function is formulated as follows:

where ¯X is the composition of X by repeating m times, and ˜X is the corrupted version of ¯X with different ratios of corruption. And tr(·) is the operator to calculate the trace of a matrix. Eq. (2) manages to minimize the original data with its transformed corrupted version so that the learned transformation is robust to noise and captures more shared discriminative information across domains. In this way, the learned transformation matrix would well leverage the disparity of two domains.

It should be noted that the single-layer low-rank coding we discussed only relies on data distributions. However, we are always accessible to labels of source domain in transfer learning. Therefore, we could pre-load these label information into model (1) where whole data with certain labels are only reconstructed by source data with the corresponding labels. Similar thought has been discussed in [Zhang et al., 2013] where image codings are guided through structured low-rank constraint. Then, we propose the final objective function:

where α is the balancing parameter and E = ¯X - W ˜X. ‖ · ‖∗ is the nuclear norm, which is a surrogate of rank() to seek a low-rank representation, whilst ‖ · ‖F is the Frobenius Norm, which aims to make the labeled representation Zl approximate to the structure matrix H. This structure term is optimized layer by layer, since most confident samples will be labeled in the target domain (refer the detail to Section 3.4). Zl is the labeled partial columns out of Z, which includes all source samples and partial target samples. We define Z = [Zl, Zu], where each column of Zu is correlated to unlabeled sample in target domain after each layer's optimization.

Discussion: Different from previous low-rank transfer learning methods [Shao et al., 2014; Ding et al., 2014], which employ the target domain to reconstruct the source one or opposite direction, we treat the transformed source domain as the dictionary and employ it to reconstruct the transformed whole data from two domains. Such constraint would optimize W, coupling source with target and also itself. Furthermore, previous ones deploy low-rank constraint on the data lying in the common subspace projection. However, our lowrank coding reconstructs the transformed data with a linear mapping learned from mDA, which would capture more discriminative and robust information shared by two domains.

Our single-layer low-rank coding (3) is developed to seek discriminative codings Z, which is guided with an iterative structured term and optimized under the transformed data via mDA [Chen et al., 2012]. In this way, single-layer low-rank coding can mitigate both the marginal and conditional distributions across two domains, and therefore, it potentially transfers knowledge from source to target and boosts the recognition performance to the target domain. Furthermore, we can stack the single-layer low-rank coding into a deep structure, where the output coding Z = [ZS,ZT ] from the previous layer would be the input of the next layer. ZS is the low-rank coding for source, while ZT is for target.

3.3 Optimization Solution

To solve Eq. (3), we first introduce a relaxing variable J and convert it to the following equivalent problem as:

which can be solved via the Augmented Lagrange Multiplier (ALM) method [Lin et al., 2010]. Since Z = [Zl,Zu],we introduce an auxiliary matrix H = [H, Zu]. We have the augmented Lagrangian function of Eq. (4) as:

where Y1 and Y2 are the two Lagrange multipliers and µ > 0 is the penalty parameter. Each variable in optimization (5) can be addressed in an iterative manner by updating J, Z,W one by one. Then, those variables are optimized in the t + 1 iteration as follows:

Update J:

which can be solved by Singular Value Thresholding (SVT) [Cai et al., 2010].

Update Z:

which is convex and has closed form solution as follows:

where Iz is the identity matrix of size nS × nS and Ψt = WtXS .

Update W :

where Rt = X - XSZt+1. Eq. (8) is convex and we can achieve its closed form solution by defining P = ¯X ˜XT and Q = ˜X ˜XT:

where the repeated number m for ¯X is expected to be ∞, giving rise to a robust denoising transformation Wt+1 learned from infinitely many copies of noisy data. Fortunately, the matrices ˆ Pt and ˆ Qt converge to their expectations when m becomes very large with the weak law of large numbers. In this way, we can derive the expected values of ˆ Pt and ˆ Qt, and calculate the corresponding mapping Wt+1 as:

where Y1,tRT t and µtRtRT t are treated as constant values when optimizing Wt+1. The expectations E[P ] and E[Q] can be derived in a similar way as in mDA [Chen et al., 2012]. The detailed optimization is outlined in Algorithm 1.

while not converged do

1. Fix others and update Jt+1 by Eq. (6);
2. Fix others and update Zt+1 by Eq. (7); 3. Fix others and update Wt+1 by Eq. (9); 4. Update two multipliers via Y1,t+1 = Y1,t + µtWt+1(X - XSZt+1); Y2,t+1 = Y2,t + µt(Zt+1 - Jt+1); 5. Update µ via µt+1 = min(ρµt, µmax); 6. Check the convergence conditions: ‖Wt+1(X -XSZt+1)‖∞ < ε, ‖Zt+1 -Jt+1‖∞ < ε. 7. t = t + 1. end while output: Z, J,W

3.4 Deep Low-Rank Coding

So far, model (3) works in a single-layer way to capture the shared information between two domains and meanwhile couple them in an iterative structure low-rank constraint.

As illustrated in our framework (Figure 1), we design a deep structure to learn more discriminative and richer information from source and target domains in a layer-wise manner. That is, we stack single-layer model (3) into multi-layer structure. Each single layer produces iteratively structured low-rank coding for both domains ZS and ZT , which would be the input of next layer. Specifically, the output from the k-1th layer ZS,k-1 and ZT ,k-1 would be the input of the kth layer, which produces ZS,k and ZT ,k. In such a layerwise scheme, DLRC would generate multi-level features for both domains and refine them from coarse to fine. The details of DLRC are shown in Algorithm 2. In the experiments, we employ five-layer features and combine them together to evaluate the final performance of our DLRC.

Algorithm 2 Algorithm of Deep Low-Rank Coding (DLRC)

Input: XS, XT , L is the number of layers,

for k = 1 to L do

1. Use Algorithm 1 to learn coding ZS,k and ZT ,k;
2. Set XS,k+1 = ZS,k and XT ,k+1 = ZT ,k;
3. Update Hk via Eq. (10);

end for

output: Low-rank codings {ZS,k, ZT ,k}, (k = 1, · · · , L).

For each layer, we need to update the iterative structure matrix H by introducing the pseudo labels of most confident samples in target domains. Suppose we label nkT samples from the target domain in the kth layer, and therefore, Hk in the kth should be an nS × (nS + nkT ) matrix. Hi,j k denotes the element of i-th row and j-th column in Hk. We seek Hi,j k through:

where yi denotes the label of xi from the labeled source and pseudo-labeled target domains. W k is the transformation matrix in the kth layer. And s(W k xi, W k xj) = exp(-‖W k xiW k xj‖2/2σ2 ) is Gaussian kernel function with σ as bandwidth (we set σ = 1 in our experiment). In this way, we can achieve the structure matrix Hk, which guides the low-rank reconstruction to minimize the conditional distribution between source and target domains. Since it is optimized layer by layer, we define it as iterative structure learning. In the experiments, we first employ the nearest neighbour classifier to predict the labels of target data using source data. Then, we label 50% target samples, which are most closest to the labeled source data according to the Euclidean distances.

3.5 Complexity Analysis

The time-cost parts of our DLRC are (1) Trace norm computation in Eq. (6); (2) Matrix multiplication and inverse in Eqs. (7) and (9).

First, Eq. (6) solved by SVD computation would cost O(n2S n) for J ∈ RnS×n . Generally, nS is the same order of magnitude with n. When n is very large, this step would be computationally expensive. But Eq. (6) can be improved to O(rn2 ) by accelerations of SVD, where r ≪ n is the rank of J. Second, Eqs. (7) and (9) both include a few matrix multiplications and a matrix inverse operation. Therefore, Eq. (7) takes (l1 + 1)O(n3 ) and Eq. (9) would take (l2 + 1)O(d3 ), where l1 and l2 are the number of multiplications for Eq. (7) and Eq. (9), respectively. In sum, the total cost of each single-layer low-rank coding is: TSLRC = O(t(rn2 + (l1 + 1)n3 + (l2 + 1)d3 )), where t is the iteration of Algorithm 1. Finally, the total cost of DLRC is LTSLRC, where L is the number of layers.

4 Experimental Results

In this section, we evaluate our proposed method on several benchmarks. We will first introduce the datasets and experimental setting. Then comparison results will be presented followed by some properties analysis and discussion.

4.1 Datasets &Experimental Setting

MRSC+VOC includes two datasets: (1) MSRC dataset 1 is provided by Microsoft Research Cambridge, which contains 4,323 images labeled by 18 classes; (2) VOC2007 dataset 2 contains 5,011 images annotated with 20 concepts. They share the following 6 semantic classes: aeroplane, bicycle, bird, car, cow, sheep. We construct MSRC+VOC by selecting all 1,269 images in MSRC and all 1,530 images in VOC2007 following [Long et al., 2013]. We uniformly rescale all images to be 256 pixels in length, and extract 128-dimensional dense SIFT (DSIFT) features.

USPS+MNIST 3 includes 10 common classes of digits from two datasets: (1) USPS dataset consists of 7,291 training images and 2,007 test images; (2) MNIST dataset has a training set of 60,000 examples and a test set of 10,000 examples. To speed up experiments, we randomly sample 1,800 images in USPS as one domain, and randomly select 2,000 images in MNIST as the other domain. We uniformly resize all images to 16×16, and represent each one by a feature vector encoding the gray-scale pixel values.

Reuters-215782 4 is a difficult text dataset with many top and subcategories. The three largest top categories are orgs, people, and place, each of which is comprised of many subcategories. For fair comparison, we adopt the preprocessed version of Reuters-21578 studied in [Gao et al., 2008].

Office+Caltech-256 5 select 10 common categories from Office dataset and Caltech-256. Office dataset has been widely adopted as the benchmark for visual domain adaptation. It has three distinct domains: Amazon, Webcam, and DSLR, including 4652 images, and 31 common categories. Caltech-256 is a standard database for object recognition, including 30,607 images and 256 categories. We apply the 800dim features by SURF+BagOfWords.

Note that the arrow "→" is the direction from "source" to "target". For example, "Webcam → DSLR" means Webcam is the source domain whilst DSLR is the target one. In the experiments, we learn five-layer features and combine them together to evaluate the final recognition performance through the nearest neighbor classifier.

4.2 Comparison Results

For MRSC+VOC and USPS+MNIST, we evaluate our algorithm by comparing with four baselines: TSC [Long et al., 2013], TCA [Pan et al., 2011], GFK [Gong et al., 2012], TJM [Long et al., 2014b]. Both two groups of datasets have two domains, therefore, we switch source and target to achieve two results for each group. The results are shown in Figure 2.

1 http://research.microsoft.com/en-

us/projects/objectclassrecognition

2 http://pascallin.ecs.soton.ac.uk/challenges/VOC/voc2007

3 http://www.cad.zju.edu.cn/home/dengcai/Data/MLData.html

4 http://learn.tsinghua.edu.cn:8080/2011310560/mlong.html

5 http://www-scf.usc.edu/∼boqinggo/domainadaptation.html

For Reuters-215782, these five baselines: TCA [Pan et al., 2011], MTrick [Zhuang et al., 2011], GTL [Long et al., 2014b], GFK [Gong et al., 2012] and ARRLS [Long et al., 2014a] are compared on six cases from three domains. The recognition results are listed in Figure 3.

For Office+Caltech-256, we compare the following baselines: SGF [Gopalan et al., 2011], LTSL [Shao et al., 2014], GFK [Gong et al., 2012], TJM [Long et al., 2014b], DASA [Fernando et al., 2013], TCA [Pan et al., 2011], mSDA [Chen et al., 2012] and GUMA [Cui et al., 2014]. We strictly follow the configuration of [Gong et al., 2012] where 20 images per category from Amazon, Caltech-256, and Webcam. Since DSLR has a small number of samples, we do not use it as source domain. Finally, we conduct 3 × 3

different groups of domain adaptation experiments. The recognition results are shown in Table 1.

Discussion: We experiment on such transfer learning scenarios, where we are only accessible to the labels of source domain. However, there are two lines. The first line, e.g. SGF, DASA, TCA, mSDA, trains in a totally unsupervised way, that is, the source label is not used in the training stage. The other line employs the source labels into training, e.g. GFK, LTSL and TSC, even introduces the pseudo labels of the target domains, e.g. TJM, ARRLS and Ours. From the results shown in Figures 2 & 3, and Table 1, we observe that our DLRC outperforms the compared baselines in most of cases under different scenarios on four benchmarks.

Compared with SGF and DASA, GFK, LTSL and TSC can achieve better results in most cases, since they incorporate the source label in order to transfer more useful knowledge to target domain. Based on this, TJM, ARRLS and Ours introduce the pseudo label of target domain into the training stage, therefore, more discriminative information can be learned in the training stage. However, mSDA in some cases performs better than other compared algorithms, which indicates that deep structure in feature learning could uncover more discriminative information across two domains. Our deep lowrank coding not only introduces the pseudo labels of the target domain, but also builds a deep feature learning framework. Therefore, our method could find plenty of rich information inside two domains and learn more helpful features for the target domains.

4.3 Properties Analysis

In this section, we evaluate on several properties of our DLRC. First, we analyze the convergence and influence of two parameters. Then, we testify the recognition performance of our DLRC with different layers. We show the evaluation results in Figure 4.

From Figure 4(a), we can observe our single-layer coding converges very fast, usually within 10-round iterations. The influence of parameters presents the recognition results on different values of two parameters in Figure 4(b). As we can see, α generates more important influence compared with λ. That means, our iterative structure term does play an important role in seeking more discriminative features for two domains. However, the larger value produces worse results. It results from the iterative structure term, which incorporates pseudo labels of target and they are not all accurate. Therefore, the larger α is, the more inaccurate information is introduced. In the experiments, we usually choose α = 10

and λ = 1. From Figure 4(c), we witness that DLRC generally achieves better performance when the layer goes deeply. That is, more discriminative information shared by two domains can be uncovered with our deep low-rank coding. In other words, features would be refined from coarse to fine in a layer-wise fashion. However, we also observe that much deeper structure would bring negative transfer and decrease the recognition performance (see case C → D in Figure 4(c)). In the experiments, we achieve five-layer features and combine them together to do the final evaluation.

5 Conclusion

In this paper, we developed a Deep Low-Rank Coding (DLRC) framework for transfer learning. First, single-layer low-rank coding guided by iterative structure learning is incorporated to align two domains, by minimizing the marginal and conditional distributions across two domains. Meanwhile, marginal denoising regularizer aims to guide the lowrank reconstruction by seeking a better transformation matrix. Finally, by stacking several single-layer low-rank transfer codings, we obtain multi-layer features with more discrimination to target domain. Experimental results on several benchmarks have demonstrated the superior of our proposed algorithm, compared with the state-of-the-art transfer learning methods.