Completely Heterogeneous Transfer Learning with Attention - What And What Not To Transfer

1 Introduction

Humans learn from heterogeneous knowledge sources and modalities, and given a novel task humans are able to make inferences by leveraging the combined knowledge base. Inspired by this observation, recent work [Moon and Carbonell, 2016] investigates a completely heterogeneous transfer learning (CHTL) scenario, where source and target tasks are heterogeneous in both feature and label spaces (e.g. document classification tasks in different languages and with different categories). In their work, CHTL is formulated as a subspace learning problem in which heterogeneous source and target knowledge are combined in a common latent space by the learned projection. To ground heterogeneous source and target label terms into a common distributed label space, they use word embeddings obtained from a language model.

However, most of the previous approaches on transfer learning do not take into account different instance-level heterogeneity within a source dataset, often leading to undesirable negative transfer. Specifically, CHTL can suffer from brute-force merge of heterogeneous sources because it does not assume explicit relations between source and target knowledge in both instance and dataset-level.

To this end, we propose a new transfer method called "Attentional Heterogeneous Transfer", with the aim of determining what to transfer and what not to transfer from heterogeneous source knowledge. The proposed joint optimization problem learns the parameters for transfer network as well as an optimized subset of source dataset, ignoring unnecessary or confounding source instances that exhibit a negative impact in learning the target task.

In addition, we propose a new joint unsupervised optimization for heterogeneous transfer network which leverages both unlabeled source and target data, leading to enhanced discriminative power in both tasks. Unsupervised training also allows for more tractable learning of deep transfer networks, whereas the previous literature was confined to linear transfer models due to a small number of labeled target data.

Note that CHTL tackles a broader range of problems than prior transfer learning approaches in that they often require parallel datasets with source-target correspondent instances (e.g. Hybrid Heterogeneous Transfer Learning (HHTL) [Zhou et al., 2014] or CCA-based methods for a multi-view learning problem [Wang et al., 2015]), and that they require either homogeneous feature spaces [Kodirov et al., 2015; Long and Wang, 2015] or label spaces [Dai et al., 2008; Duan et al., 2012; Sun et al., 2015]. We provide a comprehensive list of related work in the later section.

Our contributions are three-fold: we propose (1) a novel transfer learning algorithm that attends selectively to a subset of samples from a heterogeneous source to allow for a more tractable and accurate knowledge transfer, and (2) an unsupervised transfer with denoising auto-encoder loss unique to the heterogeneous transfer network, allowing for training deeper layers. (3) We show the efficacy of the proposed approaches on extensive simulation studies as well as a novel real-world transfer learning task.

2 Background: Completely Heterogeneous Transfer Learning (CHTL)

We begin by describing the completely heterogeneous transfer learning (CHTL) setting, where the target multiclass classification task is learned from both a target dataset and a source dataset with heterogeneous feature and label spaces. Figure 1 illustrates the overall pipeline.

2.1 Notations

Let the target task T = {XT,YT,ZT} be defined with the target samples XT = {x(i) T }NT i=1 for xT ∈ RMT , where NT is the target sample size and MT is the target feature dimension, the corresponding ground-truth labels ZT = {z(i) T }NT i=1 , where zT ∈ ZT for the categorical target label space ZT . and the parallel high-dimensional label representation YT = {y(i) T }NT i=1 for yT ∈ RME , where ME is the dimension of the embedded labels. Let LT and ULT be a set of indices of labeled and unlabeled target instances, respectively, for |LT | + |ULT | = NT . Only a few labels are available for a novel target task, thus |LT | ≪ NT . Similarly, define the heterogeneous source dataset S = {XS,YS,ZS} with XS = {x(i) S }NS i=1 for xS ∈ RMS , ZS = {z(i) S }NS i=1 for zS ∈ ZS, YS = {y(i) S }NS i=1 for yS ∈ RME , and LS for |LS| = NS (fully labeled source dataset), accordingly. The CHTL settings allow for MT ̸= MS (heterogeneous feature space) and ZT ̸= ZS (heterogeneous label space). CHTL aims at building a robust classifier for the target task (XT → ZT ), trained with {x(i) T , y(i) T , z(i) T }i∈LT as well as transferred knowledge from {x(i) S , y(i) S , z(i) S }i∈LS .

2.2 Distributed Representation for Label Embeddings

In order to relax heterogeneity between source and target label spaces, it is important to obtain a common distributed label space where all of the source and target class categories can be mapped into.

In cases where source and target class categories are represented with label terms ("names"), we can effectively encode semantic information of words in distributed representations using (1) the skip-gram based language model [Mikolov et al., 2013] trained from unsupervised text, or (2) the entity embeddings induced from a knowledge graph [Bordes et al., 2013; Wang et al., 2014; Nickel et al., 2015] with WordNet [Miller, 1995]. The obtained label term embeddings YS and YT can be used as anchors for source and target, allowing for the target model to transfer knowledge from source instances with semantically similar categories.

2.3 Transfer Network

CHTL [Moon and Carbonell, 2016] builds a transfer network with three main transformation layers: f , g, and h. g : RMS → RMC and h : RMT → RMC first project MS-dimensional source features and MT -dimensional target features into a MC-dimensional joint latent space via linear transformation, respectively. Once source and target samples are projected onto the common latent space, the transfer network maps the projected source and target samples via a shared transformation f : RMC → RME onto the embedded label space. f , g, and h are learned simultaneously by solving the joint optimization objective with hinge rank losses for both source and target. While [Moon and Carbonell, 2016] only considers linear transformation layers, we provide a more generalized objective form where f , g, and h denote mappings implemented with DNNs.

where LHR(·) is the hinge rank loss for source and target, W= {Wf ,Wg,Wh} are the learnable parameters for f , g, and h respectively, ˜y refers to the embeddings of other label terms in the source and the target label space except the ground truth label of the instance, ϵ is a fixed margin which we set as 0.1, R(W) is a weight decay regularization term, and λf , λg, λh ≥ 0 are regularization constants.

Intuitively, the weight parameters are trained to produce a higher dot product similarity between the projected source or target instance and the word embedding representation of its correct label than between the projected instance and other incorrect label term embeddings. Note that f is trained and shared by both source and target samples, thus capable of leveraging knowledge learned from a source dataset for a target task. At test time, the following label-producing nearest neighbor (1-NN) classifier is used for the target task:

where yz maps a categorical label term z into its word embeddings space. A 1-NN classifier for the source task can be defined similarly, using the projection f(g(·)) instead of f (h(·)).

3 Proposed Approaches

Figure 2 illustrates the proposed approaches.

3.1 Attentional Transfer - What AndWhat Not To Transfer

While CHTL does not assume any explicit relations between source and target tasks, we speculate that there are certain instances within the source task that are more likely to be transferable than other samples. Inspired by successes of attention mechanism from recent literature [Xu et al., 2015; Chan et al., 2015], we propose an approach that selectively transfers useful knowledge by focusing only on a subset of source knowledge while avoiding others that may have a harmful impact on target learning. Specifically, the attention mechanism learns a set of parameters that specify a weight vector over a discrete subset of data, determining its relative importance or relevance in transfer. To enhance computational tractability we first pre-cluster the source dataset into K number of clusters S1, · · · , SK, and formulate the following joint optimization problem that learns the parameters for the transfer network as well as a weight vector {αk}k=1..K:

where

where a is a learnable parameter that determines the weight for each cluster, LHR:K(Sk) is a cluster-level hinge loss for source, LSk is a set of source indices that belong to a cluster Sk, and µ is a hyperparameter that penalizes a and f for simply optimizing for the source task only. Note that f is shared by both source and target networks, and thus the choice of a affects both g and h. Essentially, the attention mechanism works as a regularization over source, suppressing the loss values for non-attended samples in knowledge transfer. In our experiments we use K-means clustering algorithm.

Optimization: We solve Eq.3 with a two-step alternating descent optimization. The first step involves optimizing for the source network parameters Wg, a,Wf while the rest are fixed, and the second step optimizes for the target network parameters Wh,Wf while others are fixed.

3.2 Unsupervised Transfer Learning with Denoising Auto-encoder

We formulate unsupervised transfer learning with the CHTL architecture for added robustness, which is especially beneficial when labeled target data is scarce. Specifically, we add denoising auto-encoders where the pathway for predictions, f , is shared and trained by both source and target through the joint subspace, thus benefiting from unlabelled source and target data. Finally, we formulate the CHTL learning problem with both supervised and unsupervised losses as follows:

where

where LAE is the denoising auto-encoder loss for both source and target data (unlabelled), g′ and h′ reconstruct input source and target respectively, and the learnable weight parameters are defined as W = {Wf ,Wg,Wh,W′g ,W′h}.

4 Empirical Evaluation

We validate the effectiveness of the proposed approaches via extensive simulations as well as a real-world application.

4.1 Baselines

Note that very few previous studies have addressed the transfer learning settings where both feature and label spaces are heterogeneous. The following baselines are considered.

• CHTL:ATT+AE (proposed approach; completely heterogeneous transfer learning (CHTL) network with attention and auto-encoder loss): the model is trained with the joint optimization problem in Eq.4.
• CHTL:ATT (CHTL with attention only): the model is trained with Eq.3. We evaluate this baseline to isolate the effectiveness of the attention mechanism.

• CHTL (CHTL without attention or auto-encoder; [Moon and Carbonell, 2016]): the model is trained with Eq.1.
• ZSL (Zero-shot learning networks with word embeddings; [Frome et al., 2013]): the model is trained for target dataset only with label embeddings YT obtained from a language model. The model thus leverages knowledge from unsupervised text corpus, and is reported to be robust for low-resourced classification tasks. We solve the following optimization problem:

where the loss function is defined as follows:

• ZSL:AE (ZSL with autoencoder loss): we add the autoencoder loss to the objective to Eq.5.
• MLP (A feedforward multi-layer perceptron): the model is trained for a target dataset only with categorical labels.

For each of the CHTL variations, we vary the number of fully connected (FC) layers (e.g. 1fc,2fc,· · · ) as well as the label embedding methods as described in Section 2.2 (word embeddings (W2V), knowledge graph-induced embeddings (G2V), and random embeddings (RAND) as a reference).

4.2 Synthetic Datasets

We generate multiple pairs of source and target synthetic datasets and evaluate the performance with average classification accuracies on target tasks. Specifically, we aim to analyze the performance of the proposed approaches with varying source-target heterogeneity at varying task difficulty.

Datasets generation process is described in Figure 3. We generate synthetic source and target datasets each with M different classes, S = {XS,YS}, and T = {XT,YT}, such that their embedded label space are heterogeneous with a controllable hyperparameter σlabel. We first generate M isotropic Gaussian distributions N(µm, σlabel) for m ∈ {1, · · · ,M}. From each distribution we draw a pair of source and target label embeddings yS,m, yT,m ∈ RME . Intuitively, source and target datasets are more heterogeneous with a higher σlabel, as the drawn pair of source and target embeddings is farther apart from each other. We then generate source and target samples each with a random projection PS ∈ RMS×ME ,PT ∈ RMT×ME as follows:

where σdiff affects the label distribution classification difficulty. We denote %LT as the percentage of target samples labeled, and assume that only a small fraction of target samples is labeled (%LT ≪ 1).

For the following experiments, we set NS = NT = 4000 (number of samples), M = 4 (number of source and target dataset classes), MS = MT = 20 (original feature dimension), ME = 15 (embedded label space dimension), K = 12 (number of attention clusters), σdiff = 0.5, σlabel ∈ {0.05, 0.1, 0.2, 0.3}, and %LT ∈ {0.005, 0.01, 0.02, 0.05}. We repeat the dataset generation process 10 times for each parameter set. We obtain 5-fold results for each dataset generation, and report the overall average accuracy in Figure 4.

Sensitivity to source-target heterogeneity: each subfigure in Figure 4 shows the performance of the baselines with varying σlabel (source-target heterogeneity). In general, CHTL baselines outperforms ZSL, but the performance degrades as heterogeneity increases. However, the attention mechanism (CHTL:ATT) is generally effective with higher source-target heterogeneity, suppressing the performance drop. Note that the performance improves in most cases when the attention mechanism is combined with the auto-encoder loss (+AE).

Sensitivity to target label Scarcity: we evaluate the tolerance of the algorithm at varying target task difficulty, measured with varying percentage of target labels given. When a small number of labels are given (Figure 4(a)), the improvement due to CHTL algorithms is weak, indicating that CHTL requires a sufficient number of target labels to build proper anchors with source knowledge. Note also that while the performance gain of CHTL algorithms begins to degrade as the target task approaches the saturation error rate (Figure 4(d)), the attention mechanism (CHTL:ATT) is more robust to this degradation and avoids negative transfer.

4.3 Hetero-lingual Text Classification

We apply the proposed methods on a hetero-lingual text classification task, where the objective is to learn a target task given a source data with heterogeneous feature space (different language) and heterogeneous labels (different categories).

Datasets: we use the RCV-1 dataset (English: 804,414 document; 116 classes) [Lewis et al., 2004], the 20 Newsgroups 1 (English: 18,846 documents; 20 classes), the Reuters Multilingual [Amini et al., 2009] (French (FR): 26,648, Spanish (SP): 12,342, German (GR): 24,039, Italian (IT): 12,342 documents; 6 classes), and the R8 2 (English: 7,674 documents; 8 classes) datasets.

Main results (Table 1): all of the CHTL variations outperform the ZSL and MLP baselines, which indicates that knowledge from heterogeneous source domain does benefit target task. In addition, the proposed approach (CHTL:2fc+ATT+AE) outperforms other baselines in most of the cases, showing that the attention mechanism (K = 40) as well as the denoising autoencoder loss improve the transfer performance (MC = 320, ME = 300, label: word embeddings). While having two fully connected layers (CHTL:2fc) does not necessarily help CHTL performance by itself due to a small number of labels available for target data, it ultimately performs better when combined with the auto-encoder loss (CHTL:2fc+ATT+AE). Note that while both ZSL and MLP do not utilize source knowledge, ZSL with word embeddings shows a huge improvement over MLP, showing that ZSL is robust to low-resourced classification tasks. ZSL benefits from autoencoder loss as well, but the improvement is not as significant as in CHTL. Most of the results parallel the simulation results with the synthetic datasets, auguring well for the generality of our proposed approach.

1 http://qwone.com/˜jason/20Newsgroups/

2 http://csmining.org/index.php/

r52-and-r8-of-reuters-21578.html

Sensitivity to attention size K (Table 2): intuitively, K ≈ NS leads to a potentially intractable training while K ≈ 1 limits the ability to attend to subsets of source dataset, and thus an optimal value of K may exist. We set K = 40 for all experiments, which yields the highest average accuracy.

Visualization of attention: Figure 5 illustrates the effectiveness of the attention mechanism with an exemplary transfer learning task (source: R8, target: GR, method: CHTL:ATT, K = 40, %LT = 0.1). The source instances that overlap with some of the target instances in the label space (near source label terms 'interest' and 'trade' and target label term 'finance') are given the most attention, which thus serve as an anchor for knowledge transfer. Some of the source instances that are far from other target instances (near source label term 'crude') are also given high attention, which may be chosen to reduce the source task loss which is averaged over the attended instances. It can be seen that other heterogeneous source instances that may yield negative impact to knowledge transfer are effectively suppressed.

Choice of label embedding methods (Table 3): While W2V and G2V embeddings result in comparable performance with no significant difference, Rand embeddings perform much poorly. This shows that the quality of label embeddings is crucial in transfer of knowledge through CHTL.

5 RelatedWork

Attention-based learning: The proposed approach is largely inspired by the attention mechanism widely adapted in the recent deep neural network literature for various applications [Xu et al., 2015; Sukhbaatar et al., 2015]. The typical approaches learn parameters for recurrent neural networks (e.g. LSTM) which during the decoding step determines a weight over annotation vectors, or a relative importance vector over discrete subsets of input. The attention mechanism can be seen as a regularization preventing overfitting during training, and in our case avoiding negative transfer.

Limited studies have investigated negative transfer, most of which propose to prevent negative effects of transfer by measuring dataset- or task-level relatedness via parameter comparison in Bayesian models [Rosenstein et al., 2005]. Our approach practically avoids instance-level negative transfer, by determining which knowledge within a source dataset to suppress or attend in learning of a transfer network.

Transfer learning with a heterogeneous label space: Zero-shot learning approaches train a model with distributed vector labels transferred from other domains, thus are more robust for unseen categories. Transfer sources include image co-occurrence statistics for image classification [Mensink et al., 2014], text embeddings learned from auxiliary text documents [Weston et al., 2011; Frome et al., 2013; Socher et al., 2013; Hendricks et al., 2016], or other class-independent similarity functions [Zhang and Saligrama, 2015].

Transfer learning with heterogeneous feature spaces: Multi-view representation learning approaches aim at learning from heterogeneous "views" (feature sets) of multi-modal parallel datasets. The previous literature in this line of work include Canonical Correlation Analysis (CCA) based methods [Dhillon et al., 2011] with an autoencoder regularization in deep nets [Wang et al., 2015], translated learning [Dai et al., 2008], Hybrid Heterogeneous Transfer Learning (HHTL) [Zhou et al., 2014], [Gupta and Ratinov, 2008], etc., all of which require source-target correspondent parallel instances. When parallel datasets are not given initially, [Zhou et al., 2016] propose an active learning scheme for iteratively finding optimal correspondences, or for text domain [Sun et al., 2015] propose to generate correspondent samples through a machine translation system despite noise from imperfect translation. The Heterogeneous Feature Augmentation (HFA) method [Duan et al., 2012] relaxes this limitation for a shared homogeneous binary classification task.

Domain adaptation with homogeneous feature and label spaces often assumes a homogeneous class conditional distribution between source and target, and aims to minimize the difference in their marginal distribution. The previous approaches include distribution analysis and instance reweighting or re-scaling [Huang et al., 2007], subspace mapping [Xiao and Guo, 2015], basis vector identification via sparse coding [Kodirov et al., 2015], or via layerwise deep adaptation [Long and Wang, 2015].

CHTL differs from the above transfer learning or domain adaptation approaches in that CHTL allows for arbitrarily heterogeneous feature and label spaces, and that it does not require instance-level correspondent datasets.

6 Conclusions

We propose a new method for completely heterogeneous transfer learning which uses the attention mechanism to determine instance-level transferability of source knowledge, as well as an unsupervised transfer loss which leads to more robust projections with deeper transfer networks. We provide both quantitative and qualitative analysis through comprehensive simulation studies as well as applications on real-world datasets. Results on synthetic datasets with varying heterogeneity and task difficulty provide new insights on the conditions and parameters in which CHTL can succeed. The proposed approach is general and thus can be applied in other domains, as indicated by the domain-free simulation results.