Source-free Domain Adaptation via Avatar Prototype Generation and Adaptation

1 Introduction

Unsupervised domain adaptation (UDA) has achieved remarkable success in many applications , such as image classification and semantic segmentation [Yan et al., 2017; Liang et al., 2019; Tang et al., 2020; Zhang et al., 2020]. The goal of UDA is to leverage a label-rich source domain to improve the model performance on an unlabeled target domain, which bypasses the dependence on laborious target data annotation. Generally, UDA methods can be divided into two categories, i.e., data-level UDA and feature-level UDA. Datalevel methods [Sankaranarayanan et al., 2018; Hoffman et al., 2018] attempt to mitigate domain shifts by image transformation between domains via generative adversarial networks [Goodfellow et al., 2014]. By contrast, feature-level methods [Ganin and Lempitsky, 2015; Wei et al., 2016] focus on alleviating domain discrepancies by learning domaininvariant feature representations. In real-world applications, however, one may only access a source trained model instead of source data due to the law of privacy protection. As a result, many existing UDA methods are incapable due to the lack of source data. Therefore, this paper considers a more practical task, called source-free UDA [Liang et al., 2020; Li et al., 2020], which seeks to adapt a well-trained source model to a target domain without using any source data.

∗Authors contributed equally.

†Corresponding author.

Due to the absence of source data as well as target domain labels, it is difficult to estimate the source domain distribution and exploit target class information for alleviating domain discrepancy as previous UDA methods do. Such a dilemma makes source-free UDA very challenging. To solve this task, existing source-free UDA methods seek to refine the source model either by generating target-style images (e.g., MA [Li et al., 2020]) or by pseudo-labeling target data (e.g., SHOT [Liang et al., 2020]). However, directly generating images from the source model can be very difficult and pseudo-labeling may lead to wrong labels due to domain shifts, both of which compromise the training procedure.

To handle the absence of source data, our motivation is to mine the hidden knowledge in the source model. By exploring the source model, we seek to generate feature prototypes of each source class and target pseudo labels for domain alignment. To this end, we propose a new Contrastive Prototype Generation and Adaptation (CPGA) method. Specifically, CPGA contains two stages: (1) Prototype generation: by exploring the classification boundary information in the source classifier, we train a prototype generator to generate source prototypes based on contrastive learning. (2) Prototype adaptation: to mitigate domain discrepancies, based on the generated feature prototypes and target pseudo labels, we develop a new contrastive prototype adaptation strategy to align each pseudo-labeled target data to the prototype with the same class. To alleviate label noise, we enhance the alignment via confidence reweighting and early learning regularization. Meanwhile, we further boost the alignment via feature clustering to make the target features more compact. In this way, we are able to well adapt the source-trained model to the unlabeled target domain even without any source data. The contributions of this paper are summarized as follows:

• In CPGA, we propose a contrastive prototype generation strategy for source-free UDA. Such a strategy can generate representative (i.e., intra-class compact and interclass separated) avatar feature prototypes for each class. The generated prototypes can be applied to help conventional UDA methods to handle source-free UDA.
• InCPGA, we also propose a robust contrastive prototype adaptation strategy for source-free UDA. Such a strategy can align each pseudo-labeled target data to the corresponding source prototype and meanwhile alleviate the issue of pseudo label noise.
• Extensive experiments on three domain adaptation benchmark datasets demonstrate the effectiveness and superiority of the proposed method.

2 RelatedWork

UnsupervisedDomainAdaptation (UDA). UDAhas been widely studied in recent years [Tang et al., 2020; Jin et al., 2020]. Most existing methods alleviate the domain discrepancy either by adding adaptation layers to match high-order moments of distributions, e.g., DDC [Tzeng et al., 2014], or by devising a domain discriminator to learn domain-invariant features in an adversarial manner, e.g., DANN [Ganin and Lempitsky, 2015] and MCD [Saito et al., 2018]. Recently, prototypical methods and contrastive learning has been introduced to UDA. For instance, TPN [Pan et al., 2019] and PAL [Hu et al., 2020] attempts to align the source and target domains based on the learned prototypical feature representations. Besides, CAN [Kang et al., 2019] and CoSCA [Dai et al., 2020] leverages contrastive learning to explicitly minimize intra-class distance and maximize inter-class distance in terms of both intra-domain and inter-domain. However, the source data may be unavailable in practice due to privacy issues, making these methods incapable.

Source-free Unsupervised Domain Adaptation. Sourcefree UDA [Kim et al., 2020] aims to adapt the source model to an unlabeled target domain without using the source data. Existing methods seek to refine the source model either by pseudo-labeling (e.g., SHOT [Liang et al., 2020]) or by generating target-style images (e.g., MA [Li et al., 2020]). However, due to the domain discrepancy, the pseudo labels can be noisy, which is ignored by SHOT. Besides, directly generating target-style images from the source model can be very difficult due to training difficulties of GANs. Very recently, BAIT [Yang et al., 2020b] proposes to use the source classifier as source anchors and use them for domain alignment. However, BAIT requires dividing target data into certain and uncertain sets via prediction entropy of source classifier, which may lead to wrong division due to domain shifts.

Compared with the above methods, we propose to generate source feature prototypes for each class instead of directly generating images. Besides, we alleviate the negative transfer brought by noisy pseudo labels through confidence reweighting and regularization.

3 ProposedMethod

3.1 ProblemDefinition

We focus on the task of source-free unsupervised domain adaptation (UDA) in this paper, where only a welltrained source model and unlabeled target data are accessible. Specifically, we consider a K-class classification task, where the source and target domains share with the same label space. We assume that the pre-trained source model consists of a feature extractor Ge and a classifier Cy. Moreover, we denote the unlabeled target domain by Dt={xi}nt i=1 , where nt is the number of target samples.

The key goal is to adapt the source model to the target domain with access to only unlabeled target data. Such a task, however, is very challenging due to the lack of source domain data and target domain annotations. Hence, conventional UDA methods requiring source data are unable to tackle this task. To address this task, we innovatively propose a Contrastive Prototype Generation and Adaptation (CPGA) method.

3.2 Overall Scheme

Inspired by that feature prototypes can represent a group of semantically similar instances [Snell et al., 2017], we explore to generate avatar feature prototypes to represent each source class and use them for class-wise domain alignment. As shown in Figure 1, the proposed CPGA consist of two stages: prototype generation and prototype adaptation.

In the stage one (Section 3.3), inspired by that the classifier of the source model contains class distribution information [Xu et al., 2020], we propose to train a class conditional generator Gg to learn such class information and generate avatar feature prototypes for each class. Meanwhile, we use the source classifier Cy to judge whether Gg generates correct feature prototypes w.r.t. classes. By training the generator Gg to confuse Cy via both cross-entropy Lce and the contrastive loss Lpcon , we are able to generate intra-class compact and inter-class separated feature prototypes. Meanwhile, to overcome the lack of target domain annotations, we resort to a self pseudo-labeling strategy to generate pseudo labels for each target data (Section 3.4).

In the stage two (Section 3.5), we adapt the source model to the target by aligning the pseudo-labeled target features to the source prototypes. Specifically, we conduct class-wise alignment through a contrastive loss Lwcon based on a domain projector Cp. Meanwhile, we devise an early learning regularization term Lelr to prevent remembering noisy pseudo labels. Lastly, to make the feature more discriminative, we further impose a neighborhood clustering loss Lnc.

The overall training procedure of CPGA can be summarized as follows:

where θg, θe and θp denotes the parameters of the generator Gg, the feature extractor Ge and the projector Cp, respectively. Moreover, λ and η are trade-off parameters to balance losses. For simplicity, we set the trade-off parameter to 1 in Eq. (1) based on our preliminary studies.

and Lnc. Note that the classifier Cy is fixed during the whole training phase.

3.3 Contrastive Prototype Generation

The absence of the source data makes UDA challenging. To handle this, we propose to generate feature prototypes for each class by exploring the class distribution information hidden in the source classifier [Xu et al., 2020]. To this end, we use the source classifier Cy to train the class conditional generator Gg. To be specific, as shown in Figure 1, given a uniform noise z∼U(0, 1) and a label y∈RK as inputs, the generator Gg first generates the feature prototype p=Gg(y, z) (More details of the generator and the generation process can be found in Supplementary). Then, the classifier Gy judges whether the generated prototype belongs to y and trains the generator via the cross entropy loss:

where p is the generated prototype and Cy(p) denotes the prediction of the classifier. In this way, the generator is capable of generating feature prototypes for each category.

However, as shown in Figure 2(a), training the generator with only the cross entropy may make the feature prototypes not well compact and prototypical. As a result, domain alignment with these prototypes may make the adapted model less discriminative, leading to insufficient performance (See Table 4). To address this, motivated by InfoNCE [van den Oord et al., 2018; Zhang et al., 2021], we further impose a contrastive loss for all generated prototypes to encourage more prototypical prototypes:

where p denotes any anchor prototype. For each anchor, we sample the positive pair k+ by randomly selecting a gener-

(a) Training with Lce

(b) Training with Lce+Lpcon

Figure 2: Visualizations of the generated feature prototypes by the generator trained with different losses. Compared to training with only the cross entropy Lce, the contrastive loss Lpcon encourages the prototypes of the same category to be more compact and those of different categories to be more separated. Better viewed in color.

ated prototype with the same category to the anchor p, and sample K-1 negative pairs k- that have diverse classes with the anchor. Here, in each training batch, we generate at least 2 prototypes for each class in the stage one. Moreover, φ(·, ·) denotes the cosine similarity and τ is a temperature factor.

As shown in Figure 2(b), by training the generator with Lce+Lpcon , the generated prototypes are more representative (i.e., intra-class compact and inter-class separated). Interestingly, we empirically observe that the inter-class cosine distance will converge closely to 1 (i.e., cosine similarity close to 0) by training with Lce+Lpcon (See Table 4), if the feature dimensions are larger than the number of classes. That is, the generated prototypes of different categories are approximatively orthometric in the high-dimensional feature space.

3.4 Pseudo Label Generation for Target Data

Domain alignment can be conducted based on the generated avatar source prototypes, However, the alignment is nontrivial due to the lack of target annotations, which makes the class-wise alignment difficult [Pei et al., 2018; Kang et al., 2019]. To address this, we generate pseudo labels based on a self-supervised pseudo-labeling strategy, proposed in [Liang et al., 2020]. To be specific, let qi=Ge(xi) denote the feature vector and let ˆ yk i =C k y (q) be the predicted probability of the classifier regarding the class k. We first attain the initial centroid for each class k by:

where nt is the number of target data. These centroids help to characterize the distribution of different categories [Liang et al., 2020]. Then, the pseudo label of the i-th target data is obtained via a nearest centroid approach:

where φ(·, ·) denotes the cosine similarity, and the pseudo label ¯ yi∈R1 is a scalar index. During the training process, we update the centroid of each class by ck= ∑nt i=1 I(¯ yi=k)qi ∑nt i=1 I(¯ yi=k) and then update pseudo labels based on Eqn. (6) in each epoch, where I(·) is the indicator function.

3.5 Contrastive Prototype Adaptation

Based on the generated prototypes and target pseudo labels, we conduct prototype adaptation to alleviate domain shifts. Here, in each training batch, we generate one prototype for each class. However, due to domain discrepancies, the pseudo labels can be quite noisy, making the adaptation difficult. To address this, we propose a new contrastive prototype adaptation strategy, which consists of three key components: (1) weighted contrastive alignment; (2) early learning regularization; (3) target neighborhood clustering.

Weighted Contrastive Alignment. Based on the pseudolabeled target data, we then conduct class-wise contrastive learning to align the target data to the corresponding source feature prototype. However, the pseudo labels may be noisy, which degrades contrastive alignment. To address this, we propose to differentiate pseudo-labeled target data and assign higher importance to the reliable ones. Motivated by [Chen et al., 2019] that reliable samples are generally more close to the class centroid, we compute the confidence weight by:

where the feature with higher similarity to the corresponding centriod will have higher importance. Then, we can conduct weighted contrastive alignment. To this end, inspired by [Chen et al., 2020], we first use a non-linear projector Cp to project the target features and source prototypes to a l2normalized contrastive feature space. Specifically, the target contrastive feature is denoted as u=Cp(q), while the prototype contrastive feature is denoted as v=Cp(p). Then, for any target feature ui as an anchor, we conduct prototype adaptation via a weighted contrastive loss:

where the positive pair v+ is the prototype with the same class to the anchor ui, while the negative pairs v- are the prototypes with different classes.

Early Learning Regularization. To further prevent the model from memorizing noise, we propose to regularize the learning process via an early learning regularizer. Since DNNs first memorize the clean samples with correct labels and then the noisy data with wrong labels [Arpit et al., 2017], the model in the "early learning" phase can be more predictable to the noisy data. Therefore, we seek to use the early predictions of each sample to regularize learning. To this end, we devise a memory bank H={h1, h2, ..., hnt } to record nonparametric predictions of each target sample, and update them based on new predictions via a momentum strategy. Formally, for the i-th sample, we predict its non-parametric prediction regarding the k-th prototype by oi,k= exp(u⊤i vk/τ) ∑Kj=1 exp(u⊤i vj/τ) , and update the momentum by:

where oi=[oi,1, ..., oi,K], and β denotes the momentum coefficient. Based on the memory bank, for the i-th data, we further train the model via an early learning regularizer Lelr, proposed in [Liu et al., 2020]:

This regularizer enforces the current prediction to be close to the prediction momentum, which helps to prevent overfitting to label noise. Note that the use of Lelr in this paper is different from [Liu et al., 2020], which focuses on classification tasks and uses parametric predictions.

Target Neighborhood Clustering. To enhance the contrastive alignment, we further resort to feature clustering to make the target features more compact. Inspired by [Saito et al., 2020] that the intra-class samples in the same domain are generally more close, we propose to close the distance between each target sample and its nearby neighbors. To this end, we maintain a memory bank Q={q1, q2, ..., qnt } to restore all target features, which are updated when new features are extracted in each iteration. Based on the bank, for the i-th sample's feature qi , we can compute its normalized similarity with any feature qj by si,j= exp(φ(qi ,qj )/τ) ∑nt l=1,l̸=i exp(φ(qi ,ql )/τ) . Motivated by that minimizing the entropy of the normalized similarity helps to learn compact features for similar data [Saito et al., 2020], we further train the extractor via a neighborhood clustering loss:

Note that the entropy minimization here does not use pseudo labels, so the learned compact target features are (to some degree) robust to pseudo label noise. We summarize the overall training scheme of CPGA in Algorithms 1, while the inference is provided in the supplementary.

Algorithm 1 Training of CPGA

Require: Unlabeled target data Dt={xi}nt i=1 ; Source model {Ge, Cy}; Training epoch E, M; Parameters η, β, τ , λ. Initialize: Projector Cp; Generator Gg.

• 1: for e = 1 →E do
• 2: Generate prototypes p based on Gg;
• 3: Compute Lce and Lpcon based on Eqns. (3) and (4);
• 4: loss.backward() based on Eqn. (1).
• 5: end for
• 6: for m= 1 →M do
• 7: Generate prototypes p for each class based on fixed Gg;
• 8: Extract target data features Ge(x) based on Ge;
• 9: Obtain target pseudo labels based on Eqn. (6);
• 10: Obtain contrastive features ht based on Cp;
• 11: Compute Lwcon, Lelr, Lnc based on Eqns. (8), (10), (11);
• 12: loss.backward() based on Eqn. (2).
• 13: end for
• 14: Output: Ge and Cy.

4 Experiments

Datasets. We conduct the experiments on three benchmark datasets: (1) Office-31 [Saenko et al., 2010] is a standard domain adaptation dataset that is made up of three distinct domains, i.e., Amazon (A), Webcam (W) and DSLR (D). Three domains share 31 categories and contain 2817, 795 and 498 samples, respectively. (2) VisDA [Peng et al., 2017] is a large-scale challenging dataset that concentrates on the 12class synthesis-to-real object recognition task. The source domain contains 152k synthetic images while the target domain has 55k real object images. (3) Office-Home [Venkateswara et al., 2017] is a medium-sized dataset, which contains four distinct domains, i.e., Artistic images (Ar), Clip Art (Cl), Product images (Pr) and Real-world images (Rw). Each of the four domains has 65 categories.

Baselines. We compare CPGA with three types of baselines: (1) source-only: ResNet [He et al., 2016]; (2) unsupervised domain adaptation with source data: MCD [Saito et al., 2018], CDAN [Long et al., 2018], TPN [Pan et al., 2019], SAFN [Xu et al., 2019], SWD [Lee et al., 2019], MDD [Zhang et al., 2019b], CAN [Kang et al., 2019], DMRL [Wu et al., 2020], BDG [Yang et al., 2020a], PAL [Hu et al., 2020], MCC [Jin et al., 2020], SRDC [Tang et al., 2020]; (3) source-free unsupervised domain adaptation: SHOT [Liang et al., 2020], PrDA [Kim et al., 2020], MA [Li et al., 2020] and BAIT [Yang et al., 2020b].

Implementation Details. We implement our method based on PyTorch 1 . For a fair comparison, we report the results of all baselines in the corresponding papers. For the network architecture, we adopt a ResNet [He et al., 2016], pre-trained on ImageNet, as the backbone of all methods. Following [Liang et al., 2020], we replace the original fully connected (FC) layer with a task-specific FC layer followed by a weight normalization layer. The projector consists of three FC layers with hidden feature dimensions of 1024, 512 and 256. We train the source model via label smoothing technique [M¨uller et al., 2019] and train CPGA using SGD optimizer. We set the learning rate and epoch to 0.01 and 40 for VisDA and to 0.001 and 400 for Office-31 and Office-Home. For hyperparameters, we set η, β, τ and batch size to 0.05, 0.9, 0.07 and 64, respectively. Besides, we set λ=7 for Office-31 and Office-home while λ=5 for VisDA. Following [Xu et al., 2020], the dimension of noise z is 100. We put more implementation details in the supplementary.

1 The source code is available: github.com/SCUT-AILab/CPGA.

4.1 Comparison with State-of-the-arts

In this section, we compare our proposed CPGA with the state-of-the-art methods. For Office-31, as shown in Table 1, the proposed CPGA achieves the best performance compared with source-free UDA methods w.r.t. the average accuracy over 6 transfer tasks. Moreover, our method shows its superiority in the task of A→D and D→A and comparable results on the other tasks. Note that even compared with the state-ofthe-art methods using source data (e.g., SRDC), our CPGA is able to obtain a competitive result as well. Besides, from Table 2, CPGA outperforms all the state-of-the-art methods w.r.t. the average accuracy (i.e., per-class accuracy) on the more challenging large-scale dataset VisDA. Specifically, CPGA gets the best accuracy in the five categories and obtains comparable results in others. Moreover, our CPGA is able to surpass the baseline methods with source data (e.g., CoSCA), which demonstrates the superiority of our proposed method. For Office-Home, we put the results in the supplementary.

4.2 Ablation Study

To evaluate the effectiveness of the proposed two modules (i.e., prototype generation and prototype adaptation) and the sensitivity of hyper-parameters, we conduct a series of ablation studies on VisDA.

Effectiveness of PrototypeGeneration. In this section, we verify the effect of our generated prototypes in the existing domain adaptation methods (e.g., DANN [Ganin and Lempitsky, 2015], ADDA [Tzeng et al., 2017] and DMAN [Zhang et al., 2019a]), which, previously, cannot solve the domain adaptation problem without source data. To this end, we introduce our prototype generation module to replace their source data-oriented parts. From Table 3, based on prototypes, the existing methods achieve competitive performance compared with the counterparts using source data, or even perform better in some tasks. It demonstrates the superiority and applicability of our prototype generation scheme.

Ablation Studies on Prototype Generation. To study the impact of our contrastive loss Lpcon , we compare the generated prototype results from models with and without Lpcon . From Table 4 2 , compared with training by cross-entropy loss Lce only, optimizing the generator via Lce+Lpcon makes the inter-class features separated (i.e., larger inter-class distance) and intra-class features compact (i.e., smaller intra-class distance). The Lpcon loss also helps to enhance the performance.

Ablation Studies on Prototype Adaptation. To investigate the losses of prototype adaptation, we show the quantitative results of the models optimized by different losses. As shown in Table 5, compared with the conventional contrastive loss Lcon, our proposed contrastive loss Lwcon achieves a more promising result on VisDA. Besides, our model has the ability to further improve the performance when introducing the losses Lelr and Lnc. When combining all the three losses (i.e., Lwcon , Lelr and Lnc), we obtain the best performance.

Sensitivity of Hyper-parameters. In this section, we evaluate the sensitivity of two hyper-parameters λ and η on VisDA via an unsupervised reverse validation strategy [Ganin et al., 2016] based on the source prototypes. For convenience, we set η = 0.05 when studying λ, and set λ = 5 when studying η. As shown in Table 6, the proposed method achieves the best performance when setting λ = 5 and η = 0.05 on Table 6: Influence of the trade-off parameter λ and η in terms of per-class accuracy (%) on VisDA. The value of λ is chosen from [1, 3, 5, 7, 9] and η is chosen from [0.001, 0.005, 0.01, 0.05, 0.1]. In each experiment, the rest of hyper-parameters are fixed.

2 Figure 2 shows the corresponding visual results of Table 4.

VisDA. The results also demonstrate that our method is nonsensitive for the hyper-parameters.

5 Conclusions

This paper has proposed a prototype generation and adaptation (namely CPGA) method for source-free UDA. Specifically, we overcome the lack of source data by generating avatar feature prototypes for each class via contrastive learning. Based on the generated prototypes, we develop a robust contrastive prototype adaptation strategy to pull the pseudolabeled target data toward the corresponding source prototypes. In this way, CPGA adapts the source model to the target domain without access to any source data. Extensive experiments verify the effectiveness and superiority of CPGA.

Acknowledgments

This work was partially supported by Key Realm R&D Program of Guangzhou (202007030007), National Natural Science Foundation of China (NSFC) 62072190, Program for Guangdong Introducing Innovative and Enterpreneurial Teams 2017ZT07X183, Fundamental Research Funds for the Central Universities D2191240, Guangdong Natural Science Foundation Doctoral Research Project (2018A030310365), International Cooperation Open Project of State Key Laboratory of Subtropical Building Science, South China University of Technology (2019ZA02).