Towards Robust Model Reuse in the Presence of Latent Domains

1 Introduction

In traditional machine learning, great efforts have been devoted to collect massive labeled data [LeCun et al., 2015], explore smart optimization techniques [Kingma and Ba, 2015] and large-scale computation power [Dean et al., 2012] to train accurate models. Nowadays, a great deal of well-trained machine learning models have been readily available to use. However, once given a new data set, the user still has to retrain a model, which obviously causes huge waste of public model resources. Meanwhile, due to privacy and security concerns, a large number of open source models usually do not allow to access raw data. Therefore, it is highly desirable to study model reuse [Zhou, 2016] which is able to reduce the learning resources for a new target task using pretrained models without the access of raw data. Model reuse has attracted much attention and many algorithms have been developed recently, e.g., [Yang et al., 2007; Ye et al., 2018; Shi and Li, 2019; Li et al., 2021].

∗This research was supported by the National Key R&D Program of China (2017YFB1001903), the NSFC (61772262) and Hikvision Cooperation Fund. Y.-F. Li is the corresponding author.

Previous model reuse studies typically assume that the instances in the target task are from one single domain [Yang et al., 2007; Ye et al., 2018; Shi and Li, 2019; Li et al., 2021]. In many real situations, however, such an assumption is hard to hold since the target task often consists of multiple latent (unknown) domains [Mancini et al., 2019]. For example, images found on the web are often a collection of many hidden domains. As shown in Figure 1, images searched from web for 'car' or 'cat' consist of multiple latent domains, such as photos, group shots, line drawings, cartoons, etc, where the domain labels are unknown beforehand. Similar situations also arise in speech recognition where the samples of the target application are often from a mixture of multiple groups of speakers (domains) [Liao, 2013]. In face recognition, the target application needs to tackle samples from multiple latent domains such as front, left and right pose [Sim et al., 2002]. In addition, it is often available to have the category label (such as 'car'), while difficult to collect the domain labels (such as 'photo', 'cartoon') [Mancini et al., 2019].

It is obvious that such setting is different to the singledomain scenario in previous model reuse studies. The model reuse methods based on single-domain scenario may no longer be feasible for multiple latent domains and may sometimes even lead to performance degeneration.

To address the above issue, in this paper, a novel method MRL (Model Reuse for Latent domains) is proposed. Our basic assumption is that the instance diversity caused by domain characteristics is helpful for the exploitation of pretrained models. Moreover, a flexible ensemble usually performs more robust than a single model. Specifically, we first construct attention transfer based on smoothness assumption [Zhou and Belkin, 2014], i.e., similar instances should have similar concept compositions within their latent domains, which motivates a similar exploitation of pre-trained models. We put it in a knowledge distillation manner and then implement knowledge transfer via a black-box prediction from pre-trained models. To achieve a reliable generalization, motivated by safe weakly supervised learning [Guo et al., 2020], we enforce the learned model to be also wellperformed on the given labeled examples in the target task, which is cast as a bi-level optimization framework with effective solutions. Empirical studies on a number of real-world data sets show that MRL achieves a clear performance gain over multiple algorithms.

2 RelatedWork

Model reuse is different to Federated Learning [Yang et al., 2019], a recently emerging area, which also provides data privacy protection. Federated learning trains a joint machine learning model on decentralized clients via iterative model aggregation between clients and control severs. Unlike federated learning, the goal of model reuse is not to build a joint model from multiple separated data sources, but to help new task with the use of pre-trained models.

Model reuse is also different to various multi-task learning studies [Pentina and Lampert, 2017], where multi-task learning aims to facilitate all the tasks via joint learning multiple tasks, and model reuse aims to facilitate the target task only via the use of pre-trained models.

Our work is related to various ensemble-based weakly supervised frameworks such as semi-supervised multi-task learning [Pentina and Lampert, 2017], semi-supervised ensemble learning [Bennett et al., 2002], and multi-source domain adaptation [Hoffman et al., 2018]. It is worth noting that these frameworks assume that target data are collected form a single domain or extract data from related tasks in a white-box way, which is not the case in our study.

There are a couple of studies proposed on mining from latent domains. Hoffman et al. [2012] proposed a constrained clustering method to discover latent domains from labeled data and build a mixture transform to implement domain adaptation. Xiong et al. [2014] proposed a squared-loss mutual information clustering model based on domain-specific local subspace estimation. Mancini et al. [2019] presented a novel deep architecture to assign a data sample to a latent domain through a side branch. There are also some studies [Mansour et al., 2008; Hoffman et al., 2018] proposing the distribution-weighted ensemble for target task composed with latent domains. However, they all learn in a scratch manner, without the exploration of pre-trained models.

There are some efforts on transferring a single pre-trained network to a new target task [Romero et al., 2015; Jang et al., 2019]. In their studies, the target samples still raise from one single domain and the pre-trained models are required to be deep models, which is clearly different to our work.

3 Proposed MRL Method

3.1 Notation and ProblemFormulation

Let (x, y) ∈ X × Y be the feature-label space where x ∈ Rd is the d-dimensional feature space and Y is the label space. Formally, training data set from the task consists of n labeled instances Dl = {(x1, y1), · · · , (xn, yn)} and m unlabeled instances Du = {xn+1, · · · , xn+m}. Usually, n is too few to derive a competitive model from scratch. m is relatively large because unlabeled data is much cheaper to obtain. Note that the target samples does not have any domain label.

Suppose there are K pre-trained models {h1, · · · , hK} each corresponds to a latent domain, built on the same feature-label space X × Y with different sources. Formally, there is a small constant error rate ϵ > 0 with a certain loss function L on their task distribution Di: ∀i ∈ [K], E(x,y)∈Di [L(hi(x), y)] ≤ ϵ. The goal of model reuse is to learn a model h(x; θ) : {X;Θ} → Y parameterized by θ ∈ Θ from training data to minimize the generalization risk RT (h) = E(x,y)∈DT [L(h(x; θ), y)] where DT denotes the distribution of the target task. Generally, L is convex and bounded by a certain constant M. Without loss of generality, we assume DT = ∑Kk=1 λkDk with unknown λ = [λ1, . . . , λK] [Mansour et al., 2008]According to structural risk minimization, it is formalized as following:

where x ∈ {xi}n+m i=1 and Ω(x; θ, {hk}Kk=1 ) refers to the regularization term with the help of pre-trained models, which is the key to model reuse. Ω could be realized in different ways, like output consistency [Hinton et al., 2015], representation consistency [Romero et al., 2015], etc. In this paper, we choose output consistency to instantiate Ω, as it is good at coping with the probability output on label space, no matter for neural networks or trees. Specifically, Ω(x; θ, {hk}Kk=1 ) could be rewritten as D[h(x; θ),R(x)], where R represents a specific reuse strategy and D represents distance measure, like KL-Divergence, L2 distance, etc.

3.2 Consistent Reuse and Its Deficiencies

A straightforward and effective method to utilize pre-trained models is to consider a linear ensemble of existing models with weights w for the target task, i.e.,

where W is typically a convex set as {w | wi ≥ 0, ∑i wi = 1}. The weights could be solved in various ways, such as performance error [Murugesan et al., 2016; Shi and Li, 2019; Zhao et al., 2020] , min-max game [Li et al., 2021], empirical discrepancy [Pentina and Lampert, 2017] and maximum mean discrepancy [Duan et al., 2012], etc. These methods rely on a consistent scheme that utilizes the target data in a same manner. We call them consistent reuse in this paper. Now, we present the robustness analysis of consistent reuse on DT .

Theorem 1. Note that DT = ∑Ki=1 λiDi, the upper bound of consistent reuse Rw satisfies, for α > 1,

where dα(D‖D′ ) = eDα(D‖D′ ) denotes the exponential of the R´enyi Divergence of two distributions D and D′ . In the worst case ∀i ∈ [K], dα(Di || DT ) →∞, α → 1, the upper bound could be tailored as:

Theorem 1 indicates that consistent reuse is not effective to distribution DT , particularly there is no guarantee for consistent reuse in the worst case. Figure 2 illustrates our observation on synthetic data. When we adapt previous accurate models from multiple sources to the target distribution DT = 1K ∑Kk=1 Dk, consistent reuse 1K ∑Kk=1 hk is poor because of the large divergence between any individual source distribution Di and the target distribution DT . As a result, each individual hk performs well on one part of data distribution while performs poorly on the whole distribution.

To address such critical limitation, we propose an approach to explore the instance diversity in the target distribution associated with latent domains. In the following subsections, attention reuse, which is designed to explore instance diversity and exploit pre-trained models, is first introduced and then the optimization scheme.

3.3 Attention Reuse

As human being, one often asks different questions to different teachers/experts, e.g., ask the maths teacher for theoretical questions and the engineering teacher for practical questions. Although one single teacher may not be able to handle all the questions, one can ask the corresponding teacher for diverse queries. In this section, we simulate such a human pattern to build an attention mechanism to implement a flexible knowledge transfer.

To achieve it, one first needs to properly specify how attention is defined w.r.t. given pre-trained models {hk}Kk=1 . To that end, here we consider attention as the model-wise confidence w for vary queries. Based on smoothness assumption [Zhou and Belkin, 2014], that is, similar instances should have similar concept compositions within their latent domains, we further argue if model hk could correctly predict x, it is likely to correctly predict instances similar to x. Thus, we construct an attention reuse Rφ based on an instance-aware function g : X →W parameterized by φ.

We construct a description of the marginal distribution for each model, which is named as Prototype. The final prediction is then computed as a weighted sum of the predictions {hk(x)}Kk=1 , where the weight assigned to each prediction hk(x) is computed by a compatibility function of the query with its corresponding Prototype, as illustrated in Figure 3. In practice, we use a feature extractor ψ : X → Rp to obtain a low-dimensional embedding of input, and build a dot-product attention [Vaswani et al., 2017] on ψ(x). The K prototypes [Pk]Kk=1 ∈ Rp×k and predictions [hk(xi)]Kk=1 are packed together into matrices P and H. We compute the attention output for query x as:

where ψ could be instantiated as convolution networks, like a resnet-backbone pre-trained on Imagenet. Notice that the attention mechanism could be optimized via Eq. 4 when we only have category labels.

Compared to consistent reuse, attention reuse favors a better exploration of the target data with latent domains through feeding various instances to corresponding models. It owns better robustness guarantees based on [Mansour et al., 2008].

Algorithm 1 The proposed MRL method

Input: Training data {(xi, yi)}ni=1 and {xi}n+m i=n+1 , predictions H, learning rate ηθ and ηφ, maximal iteration T . Output: Target model h(θT )

• 1: Initialize θ0 through supervised learning.
• 2: Initialize φ0 on labeled data via Eq. 4.
• 3: for t = 0 to T -1 do
• 4: Sample data from {(xi, yi)}ni=1 and {xi}n+m i=n+1
• 5: Compute inner loss Linner(θt, φt)
• 6: Update θt+1 = θt - ηθ∇θLinner(θt, φt)
• 7: Update φt+1 = φt - ηφ∇φLouter(θt+1)
• 8: end for
• 9: return h(θT )

Theorem 2 (Robustness). Note that DT = ∑Ki=1 λiDi, the upper bound of attention reuse in the worst case satisfies:

Compared to Theorem 1, Theorem 2 indicates that the proposed attention reuse is provably more robust than previous consistent reuse. The experimental results also confirmed our theoretical findings.

3.4 Optimization and Analysis

We denote the target model derived under Eq. 1 and φ as ˆ θ(φ). Remind that the goal of model reuse is to maximize the generalization E(x,y)∈DT [L(h(x; ˆ θ(φ)), y)] of the target model ˆ θ(φ). In practice, we derive φ through the empirical risk on labeled data as ∑ni=1 L(h(xi; ˆ θ(φ)), yi). To simplify the notation, we denote ˆ θ(φ) as ˆ θ. The overall consideration is cast under a bi-level optimization scheme [Colson et al., 2007].

where Ωφ (xi; θ, H) = D[h(x; θ), Rφ(x)].

There are various algorithms solving the bi-level optimization in Eq. (5), such as single-level reduction, gradient descent and evolutionary algorithms [Sinha et al., 2018]. We choose an efficient alternating optimization method [Jang et al., 2019] to solve it. The overall optimization procedure is summarized in Algorithm 1. We first optimize h(θ) and φ on labeled data and employ them as the initialization, for bilevel optimization. In inner optimization, we update θ with reuse strategy Rφ and predictions H from previous models. In outer optimization, we can obtain the outer objective, and update φ through bi-level gradient ∇φLouter(θt+1), which can be calculated by implicit function theorem and chain rule [Shu et al., 2019].

Convergence

The convergence analysis of bi-level optimization have been well-studied in previous studies [Ren et al., 2018; Shu et al.,

2019]. Suppose the loss function is ζ-Lipschitz smooth and the gradient in inner/outer optimization is bounded by δ. Our method could achieve E[||∇Louter(θt)||2 ] ≤ ϵT in O(1/ϵ2T ).

Generalization

We further analyze the generalization risk of MRL based on [Zhao et al., 2019; Guo et al., 2020] to better understand the comparison w.r.t. learning from scratch.

Theorem 3 (Generalization). Assume L is ζ-Lipschitz continuous w.r.t. φ. Let φ ∈ Rd′ (d′ = K ∗ p) be the parameters in a unit ball, and n be the labeled data size. Let φ∗ = argmaxφ∈Rd′ RT (ˆ θ(φ)) be the optimal parameter in the unit ball, and ˆ φ be the empirical optima among a candidate set A. With probability at least 1 - δ we have,

It is noteworthy that supervised learning which optimizes high-dimensional (d) parameters θ, achieves the optimal weight in the order O(√d ln(d) ln(n)/n) [ShalevShwartz and Ben-David, 2014]. By contrast, we learn a lowdimensional (d′ ≪ d) attention module φ via bi-level optimization, sharing a order O(√d′ ln(n)/n), as established in Theorem 3, favors a better order than learning from scratch.

In this work, we forge a connection between knowledge distillation and teacher-student semi-supervised learning [Qi and Luo, 2019] to build a target model, decoupled with pretrained models, i.e., does not need to recall pre-trained models for new queries.

4 Empirical Study

4.1 Experimental Setup

To validate our method, we perform experiments on diverse tasks, including Digital Recognition, Attribute Classification and Face Recognition. All competing methods are implemented on PyTorch 1 . The output consistency D is instantiated with L2 distance. The hyper-parameters are adjusted by the validation set for all methods.

Learn from Scratch Besides supervised learning, we have compared three popular teacher-student SSL methods in deep learning community, i.e., Pseudo Label (PL) [Lee, 2013], Temporal Ensembling (TE) [Laine and Aila, 2017] and Virtual Adversarial Training (VAT) [Miyato et al., 2018]. PL teaches unlabeled data via entropy regularization, TE tracks a model ensemble over time to have a better teacher model, and VAT generates adversarial noise on input to construct a robust teacher. They utilize the consistency of unlabeled data without much prior knowledge, showing promising performance.

Learn with Reuse We have also compared the effect of model reuse with the best single teacher selected from labeled data, as well as the consistent reuse. Without loss of generality, we implement black-box knowledge transfer based on distillation [Hinton et al., 2015]. We denote them as MR-BS (Model Reuse with Best Single source) and MR-CR (Model Reuse with Consistent Reuse), respectively.

1 https://pytorch.org/

4.2 Tasks on Digital Recognition

We begin our study with a generalization of digital recognition task, which consists of three digital recognition datasets: SVHN, MNIST, and USPS. Generally, we use the full recommended training sets per domain to learn the source models, and select test images from 3 datasets to construct the target tasks. For consistency, we resize these images to 28x28, and convert the images from SVHN to gray scale.

During reuse, we take 10% samples for validation and 40% samples for testing. Besides, we extract 500, 100, 100 labeled samples from SVHN, MNIST and USPS respectively, and the remaining samples as unlabeled data. Specifically, we train a CNN with 3 convolution layers and take the output of convolution as the representation (512-dim) to derive attention.

Table 1 shows the average accuracy of 5 random splits on 7 different target tasks composed with different domains. For examples, "MSU" means the target task composed with three domains, "MS" means the target task composed with domains MNIST and SVHN. It can be found that the consistent reuse method is often inferior to the direct semi-supervised learning methods, and its performance is not robust. In contrast, our method consistently outperforms semi-supervised learning methods and baselines, and shows stronger robustness. Particularly in the average, our method has a 13.1% performance gain over supervised baseline, which is clearly better than the consistency method (3.0%) and SSL (3.4%).

4.3 Tasks on Attribute Classification

Our second set of experiments is based on the Animals with Attributes 2 dataset 2 , which contains 37,322 images of 50 animal classes. These animals are aligned to 85 binary attributes, e.g. color, habitat, via a class-attribute binary matrix, indicating whether an animal possesses each feature.

2 https://cvml.ist.ac.at/AwA2/

We select 4 attributes about habitat {plains, mountains, water, tree}, to simulate the different sources and construct multiple tasks of identifying whether the animal on a given image possesses a certain attribute or not. Firstly, we split 27,322 samples and divided them into these 4 sources according to the habitats. For example, horses belong to plains, dolphins belong to water, and pandas belong to tree. Then, four models were built on these sources respectively. Such a setting is consistent with data sources for different geographic locations in real-world. The target task is selected from some advanced attributes, for example, to determine whether the animal has spots, whether it is smart, etc. For the remaining 10,000 samples, we use 50% as training set, 10% examples as validation set and take the others for testing. In addition, we construct an {2048, 200, 2} MLP as the target model, and use the 2048-dimensional features, taken by convolution of ILSVRC pre-trained ResNet101 [Xian et al., 2018], as the input of our MLP and attention module. The average results (with 200 labeled samples) under 10 times random splits are reported. From Table 1, it can be found that the consistent reuse method does not work well, i.e., it is usually inferior to the direct semi-supervised learning methods. In contrast, our method behaves much more robust, i.e., it consistently outperforms semi-supervised learning methods and baselines over all the subtasks.

A key factor of model reuse is to study the effectiveness when the labeled data is limited. In these cases, model reuse would be more desriable. We sample various labeled samples and compare the performance of Supervised, VAT, MRCR and MRL on "spots" task. As shown in Figure 4, MRL consistent achieves better performance gain over compared methods, especially when labeled samples are few.

To further evaluate the effectiveness of attention module, we visualize the average value of similarity learned on different prototypes in the upper part of Figure 5. The lower part has shown the ground truth of habitat. We could find that our attention module could reasonablely learn the relationship between instances and habitats (latent domains).

4.4 Tasks on Face Recognition

Finally, we evaluate our method on the CMU Multi-PIE dataset [Sim et al., 2002], which is a facial expression dataset. In this experiment, five domains generated from Multi-PIE (each corresponding to a distinct pose) from 68 individuals. Specifically, five subsets, i.e., PIE05 (left), PIE07 (upward), PIE09 (downward), PIE27 (front), PIE29 (right) are constructed, and the face images in each subset are taken under different illumination and expression conditions. These subsets 3 are based on SURF features and the dimension of features is 1024.

3 https://github.com/jindongwang/transferlearning/blob/master/ data/dataset.md

We extract 800 samples per domain to construct the target task, the remaining samples are used as the source data to build pre-trained models separately. Due to the limited number of samples, we use 680 samples (2 samples per category per domain) as testing set, and 340 samples (1 sample per category per domain) as validation set. The rest of the target data is regarded as unlabeled data pool. In addition, we use Logistic Regression as the base model, and use SURF feature as the input of attention module. The results on one-shot (1 sample per category per domain) and two-shot cases in Table 2 further demonstrate the robustness of our proposal.

5 Conclusion

In this paper, we study the model reuse for latent domains problem where the target data are composed with latent or unknown domains. This is a new kind of model reuse which to the best of our knowledge, has not been thoroughly studied. In this paper we propose a novel MRL method. Both domain characteristics and pre-trained models are considered for the exploration of instances in the target task. The learned model is enforce to be well-performed on two different objectives, which is cast as a bi-level optimization with effective solutions in a reliable generalization. Empirical results verify our effectiveness and robustness.