Joint Partial Optimal Transport for Open Set Domain Adaptation

1 Introduction

Deep learning recently has improved the progress of diverse computer vision tasks such as image classification, object detection and semantic segmentation with the help of largescale labeled datasets which is time and labor consuming to collect. However, it is often highly difficult to achieve satisfying performance when transferring knowledge from labelrich domain (i.e., source domain) to label-scarce domain (i.e., target domain), especially when there are unknown samples in the target domain. The characteristics of large scale unlabeled data in the target domain can be different from the labeled source domain, resulted in misclassification.

∗Corresponding authors: {rux, zhangyin98}@zju.edu.cn

An underlying assumption in domain adaptation is that samples in the target domain necessarily belong to the classes observed in the source domain, from which we train the classifier. Since the target classes are often larger than training classes in real-world settings, considering the open set domain problem could lead to more practical results and applications. Under this regime, the target domain consists of subset of classes of the source domain and along with extra unknown classes. The main task is seeking to classify data of known classes correctly, and categorize the extra classes as "unknown". Lack of openness adaptation is a major critique against OSPB [Saito et al., 2018].

We solve the problems above by a strategy based on the optimal transport framework. Optimal transport has been applied to closed set domain adaptation recently [S. and S., 2016; Luo et al., 2017; Damodaran et al., 2018]. Damodaran et al. recognized the strength of Convolutional Neural Network (CNN) to obtain a scalable solution by learning jointly the feature embedding between the two domains and the classifier in a single CNN framework. All previous OT based methods align the whole source and target domains and perform badly for OSDA since data of unknown classes in the target domain can make performance of domain adaptation model even inferior to a model without adaptation. Such phenomenon is known as negative transfer [Pan and Yang, 2010]. Unknown samples in the target domain need to be excluded in the transfer to avoid negative transfer. To this end, we propose a more flexible partial optimal transport framework that allows for transportation in only well-matched pairs of samples. We formulate the problem as transporting a fraction of the "well-matched" mass of source domain onto the target domain minimizing an overall displacement cost. We eliminate the far-fetched pairings that cause negative transfer in the global optimal coupling to obtain a partial optimal coupling. Besides the regular displacement loss, the overall displacement loss includes the loss for displacing to the unknown class applied to the fraction of mass not transferred. As a bonus, our method naturally adapts to different levels of openness of the target domain, because fraction size selected from global coupling is data-adaptive.

Furthermore, we want to choose the feature representation that leads to better generalization and robust performance. For this goal, we include dispersion penalty for each class so that samples are concentrated around the means of their classes and therefore lie far from the decision boundary; we add to the objective function the joint discriminative prototypical compactness loss, which provides a way to estimate the mean and manipulate (shrink) the covariance of each class (including the unknown class). We firstly estimate the mean and variance for the unknown class from the residuals of our partial optimal transport which was a challenge for previous researchers as we are blind about the structure of unknown classes. Experimental results show that JPOT works stably in various datasets and outperforms existing methods.

2 RelatedWork

ClosedSet DomainAdaptation (CSDA). Currently closed set domain adaptation methods [Tzeng et al., 2014; Long et al., 2015; Long et al., 2016] minimized the feature distribution discrepancy to alleviate performance degradation in both source and target domains. Zellinger et al. [Zellinger et al., 2017] achieved domain adaption by Central Moment Discrepancy (CMD). Contrastive Adaptation Network (CAN) [Kang et al., 2019] optimizes a new metric that explicitly models the intra-class domain discrepancy and the inter-class domain discrepancy. Transferrable Prototypical Networks (TPN) [Pan et al., 2019] exploits the prototypical distance [Snell et al., 2017] to avoid misclassification.

Open Set Domain Adaptation. OSDA differs fromCSDA by extra unknown classes in the target domain but not in the source domain. Recently, researchers are concentrating on open set recognition, intending to recognize "unknown" samples in the target domain during testing [Ganin and Lempitsky, 2015; Saito et al., 2018]. Separate to Adapt (STA) network was proposed by [Liu et al., 2019] to adapt different openness between source and target classes with coarse-tofine weighting mechanism. Universal Adaptation Network [You et al., 2019] was proposed to quantify sample-level transferability and recognize the "unknown" samples. However, all of them failed to solve the problem that target domain samples were distributed near the decision boundary learned from the source domain, leading to degraded performance.

Optimal Transport on Domain Adaptation. Optimal transport has been applied in domain adaptation to align the representations between the source domain and target domain [Courty et al., 2016; Yan et al., 2018]. As mentioned above, the coupling γ [Courty et al., 2017] was applied to transport the source samples to the target domain by an estimated mapping. Deep Joint Optimal Transport (DeepJDOT) [Damodaran et al., 2018] applied the coupling matrix γ to transport the source samples to the target domain in discriminative feature space, achieving high accuracy on many transfer tasks. However, the discriminative representation of "unknown" samples in the target domain has not been considered in previous approaches that causes mismatching.

3 Methodology

Following the settings of OSDA [Liu et al., 2019], we define a source domain Ds = {(xsi , ys i )}ns i=1 with ns labeled samples and a target domain Dt = {(xti )}nt i=1 with nt unlabeled samples. The label space in the source domain is defined as C s , which is a subspace of target-domain label space C t . The target domain further contains additional classes C tunk , which are all referred as "unknown" since these classes are absent in the source domain, i.e. C t = C s ∪ C tunk . Data from source and target domains can have different probability distributions p and q respectively (p ̸= q). In OSDA, we further observe that p ̸= qCs, where qCs represents the probability distribution of the target data belonging to shared classes C s .

Following [Saito et al., 2018; Chen et al., 2019], we adopt the two-stream CNN architecture with shared weights. We propose joint partial optimal transport (JPOT) for open set domain adaptation, which is an end-to-end method that learns a feature generator Gf(·) and a classifier Gy(·) to separate known and unknown classes in the target domain, as illustrated in Figure 2.

3.1 Partial Optimal Transport

Optimal transport for domain adaptation performs the alignment of the sample representations between the source domain and target domain [Courty et al., 2016]. However, existing optimal transport strategies fail to address the problem caused by the pair-wise transport error in pairing unknown target samples to known source samples, resulting in negative transfer of unknown classes in the target domain. Despite similarities with the formulation [Damodaran et al., 2018], we propose partial optimal transport strategy exploiting global coupling matrix γ given by joint probability distribution, which aims to identify the most correlated features of shared classes between the source and target domains. Moreover, we utilize global optimal matrix to capture distinct features of unknown class samples in the target domain and to realize inter-class separability of unknown vs known classes. The main procedure of JPOT is shown in the Figure 1.

The optimization of partial optimal transport is based on partial Kantorovich problem [Angenent et al., 2003] seeking for a general coupling γ ∈ X(Ds , Dt ) between Ds and Dt :

where X(Ds ,Dt ) denotes the probability distribution between Ds and Dt . The cost function matrix M(xs , xt ) = ||Gf(xs ) - Gf(xt )||k denotes the cost to move probability mass from xs to xt , where k = 2 [Damodaran et al., 2018]. The discrete formulation can be expressed as

where γ ∗ ∈ RN×N is the ideal coupling matrix between the source and target domains, representing as a joint probability measure with source data xs and target data xt . N is the batchsize. The elements of each row of γ ∗ are all zeros except one equals 1. 〈·, ·〉F is the Frobenius dot product. To separate data of unknown class from known classes in the target domain, we utilize the mean cost of optimal transport matrix ρ to measure the similarity between each target sample and each source sample

where N denotes the number of source samples which equals that of target samples. If the coupling pair distance is bigger than the mean cost ρ, it indicates that this coupling pair is nontransferable. The partial optimal transport for open set domain alignment can be defined as

where ˆ γ ∗ represents partial coupling matrix for known classes of both domains. sgn(x) = 1 when x > 0, and equals

-1 otherwise. Thus, we rank the similarity ρj for all target samples and select samples with highest/lowest similarity to align/separate. The loss for known classes in the source and target domains can be defined as

which means that the best matched pairs should be pulled each other closely. Meanwhile, with samples "labeled as" unknown class by partial coupling matrix, they should leave away from known classes to avoid negative transfer

where ∆γ ∗ i,j = γ ∗ i,j - ˆ γ ∗ i,j denotes the residual set of the partial optimal transport ˆ γ ∗ i,j. η denotes as a constant and equals to 0.1 in the experiment. Therefore the partial optimal transport loss Lp is given as

3.2 Joint Discriminative Prototypical Compactness

Inspired by previous work [Herath et al., 2019], exploiting discriminative statistics information on two domains will enhance the transfer performance. When the feature distribution, Ps of domain Ds , is parameterized with the mean µsz and the covariance Σsz

where z = {1, 2, · · · , k} denotes the corresponding shared known classes, k denotes the number of classes in Ds , δ(x) = 1 when x is True, and δ(x) = 0 when x is False. The variance vanishes when there is only one sample in the batch. Due to the lack of reliable labels in the target domain, it is difficult to measure the unknown samples in the target domain distribution Pt. The most difficult task is to find the mean and variance of unknown samples. According to the partial optimal transport, we provide unknown index Uj to measure the probability of sample xtj to be in the Unknown class.

The number of the unknown class samples can be given as

The mean µunk and the covariance Σunk can be described as

To quantify spatial prototypical information in both domains, we define µA = {µs1 ,µs2 , · · · ,µsk ,µtunk } and ΣA = {Σs1 ,Σs2 , · · · ,Σsk ,Σtunk } for computation. Therefore µA ∈ R(k+1)×d and ΣA ∈ R(k+1)×d×d . d is the number of output neurons in the bottleneck layer. The spatial prototypical matrix D ∈ R(k+1)×N is defined as

where d(Gf (xtj ), µAz ) is the distance between the target sample Gf(xtj ) and z-th class center µAz . Therefore, the Mahalanobis distance formulation of d(Gf (xtj ), µAz ) can be defined as

The joint discriminative prototypical compactness loss Ld is defined as

The Ldc denotes the centroid clustering loss in the source class domain

The Ldp denotes the prototypical compactness loss for the target samples

where F(·, ·) denotes the cross-entropy loss. Furthermore, we introduce an additional loss Lcls to minimize the probability of source samples to be misclassified as unknown class.

In conclusion, the total loss of our JPOT model is

where the α and β denote the trade-off parameters.

To calculate the statistic information, we adopt the incremental learning strategy as follows:

where µA(t) accu. and ΣA(t) accu. denote the accumulation results of the mean and variance, 0 ≤ m ≤ 1 is the momentum hyperparameter for the accumulation.

4 Experiments

Now we evaluate our method with state-of-the-art domain adaptation approaches on several benchmark datasets.

4.1 Setup

Digits contains three standard digit classification datasets: MNIST [Lecun et al., 1998], USPS [Hull, 2002], and SVHN [Netzer et al., 2011]. Each dataset consists of 10 classes of digits, ranging from 0 to 9. Following the same evaluation protocol of [Saito et al., 2018], we construct three open set domain adaptation tasks and report adaptation results on the test sets: SVHN→MNIST (one task) and MNIST↔USPS (two tasks).

Office-31 [Saenko et al., 2010] is a standard dataset in computer vision for domain adaptation which contains 4652 images from 31 categories with three domains: Amazon (A), Webcam (W) and DSLR (D). According to previous work [Liu et al., 2019], we adopt the same set of known classes and unknown classes in the target domain and evaluate all methods

on following four challenging settings: A↔W and A↔D.

Office-Home [Venkateswara et al., 2017] is a more challenging domain adaptation dataset which consists of around 15500 images from 65 object classes in 4 distinct domains: Aetistic (Ar), Clipart (Cl), Product (Pr), and Real-Word (Rw). We select (in alphabetic order) the first 25 classes as known classes shared by the source and target domains [Saito et al., 2018]. The 26-65 classes are regarded as the unknown class. We report performances of all the 12 adaptation tasks to enable thorough evaluations: Ar↔Cl, Ar↔Pr, Ar↔Rw, Cl↔Pr, Cl↔Rw, and Pr↔Rw.

Compared Approaches. We mainly compare our proposal with several open set recognition, domain adaptation and open set domain adaptation methods as previous work [Liu et al., 2019]: Open Set SVM(OSVM) [Jain et al., 2014], DANN [Ganin and Lempitsky, 2015], RTN [Long et al., 2016], OpenMAX [Bendale and Boult, 2016], MMD+OSVM, DANN+OSVM, OSBP [Saito et al., 2018] and STA [Liu et al., 2019]. MMD+OSVM and DANN+OSVM are two variants of OSVM that incorporate Maximum Mean Discrepancy [Tzeng et al., 2014] and domain adversarial network [Ganin and Lempitsky, 2015] in OSVM. In our experiments, we compare the average accuracy of each method on five random experiments.

Evaluation Metrics. Following previous works [Saito et al., 2018], we employ four evaluation metrics for open set domain adaptation: OS: the accuracy averaged over all the classes including the unknown class as one class; OS*: the accuracy averaged only on known classes; ALL: the accuracy of all samples; UNK: the accuracy of unknown instances.

4.2 Implementation Details

For the experiments on Office-31 and Office-Home, our proposal and the compared approaches are both trained by finetuning with ResNet-50 [He et al., 2016] pre-trained on ImageNet. And we adopt LeNet [Lecun et al., 1998] to investigate the efficacy of our framework for the experiments on Digits datasets. We use all labeled source examples and unlabeled examples for training.

All the mentioned deep learning methods are trained with Adam optimizer. And the model is trained on 256-sized batches totally with N = 128 samples from each domain. We use mini-batch SGD with momentum to 0.9 and the same learning rate strategy in [Saito et al., 2018]. Following previous work [Herath et al., 2019], m is set as 0.4 in the whole experiment. Note that the parameter η is set as 0.1. As for tradeoff hyper-parameters α and β, we select α = 0.02 and β = 0.05 for all transfer tasks.

4.3 Result and Discussion

The classification accuracy results on the Office-31 dataset for open set domain adaptation based on ResNet-50 are shown in Table 1. For fair comparison, all results of compared approaches are directly reported from their original papers. The JPOT exceeds the performance of all previous methods on most transfer tasks. It is worth noting that our proposal improves the classification accuracies substantially on hard transfer tasks, e.g. A→W and W→A, and achieves comparable classification performance on easy transfer tasks, e.g. D→W and W→D. In particular, JPOT produces higher OS than OS*, verifying that JPOT is robust to open set domain adaptation scenarios. Besides, some close set domain adaptation methods perform worse than ResNet on a few tasks since these methods suffer from negative transfer caused by mismatching unknown classes in the target domain to known classes in the source domain. JPOT aligns the discriminative representations of shared class centers in both source and target domains by center-based optimal transport strategy, and achieves intra-class compactness and inter-class separability.

Results on the twelve tasks of Office-Home dataset are shown in Table 2. Due to the large domain gap between the source and target domains, we observe that previous open set domain adaptation approaches obtain poor performance on some tasks. The JPOT approach outperforms the comparison methods on most transfer tasks. The encouraging results suggest that our proposal alleviates the negative transfer issue brought by the unknown class in the target domain, and also highlight the importance of center-based optimal transport strategy in open set domain adaptation.

We further compare JPOT with previous approaches on the Digits dataset, as reported in Table 3. In contrast to Office-31 dataset, Digits dataset has a much larger domain size. JPOT overpasses all comparison methods with different metrics on most transfer tasks. Note that, SVHN dataset contains significant variations (in scale, slanting, blurring and rotation) and confounding information from background that makes large domain gap on task SVHN→MNIST. JPOT achieves better performance than STA on task SVHN→MNIST. In particular, JPOT almost achieves the state-of-the-art performance with the UNK accuracy in all transfer tasks, which suggests that JPOT is robust to large domain gap and able to learn more transferable representations for open set domain adaptation.

4.4 Ablation Study

To tooth apart the separate contributions of center-based optimal transport strategy and discriminative domain alignment, we compare JPOT with OSBP [Saito et al., 2018], DeepJDOT [Damodaran et al., 2018] and a variant of JPOT using the t-SNE embeddings [Donahue et al., 2014] of the lastlayer features on transfer task MNIST→USPS in Figures 3(a)-3(h). (1) In Figures 3(a) and 3(e), we observe that in the OSBP model, scattered samples of unknown and known classes in the target domain are close or even mixed together and the domains are not well aligned, which may cause negative transfer. Therefore the OSBP output classification is of low confidence. (2) In Figures 3(b) and 3(f), although the original DeepJDOT computes the global optimal transport solution, the target unknown samples still be mistaken transported to the corresponding source domain. Apparently the unknown samples mixes up with the known samples which leads to terrible prediction results. (3) As shown in Figures 3(c) and 3(g), compared with OSBP and DeepJDOT, JPOTP (Lcls + αLp) with partial optimal transport strategy and without joint discriminative prototypical compactness loss, achieves better domain alignment with high classification performance meanwhile avoid negative transfer. Due to the fact that partial optimal transport exploits the most likely to be the unknown samples, the unknown target samples will not fully mix up with the known samples. (4) As shown in Figures 3(d) and 3(h), JPOT achieves intra-class compactness and interclass separability, and also improves with a large room over JPOT-P, validating the complement of discriminative feature learning with the corporation of partial optimal transport.

In Table 4, further ablation tests have been carried out on variations of JPOTs including JPOT-T (Lcls + αLkno o ), JPOT-S (Lcls+αLp+βLdc), and JPOT-P. (1) While JPOTT still works well on detecting unknown instances (87.4% of accuracy), the performance falls behind other JPOT variations. Without the additional loss Lunk o , JPOT-T lacks driving force to push unknown samples away from the known samples in the feature space, resulted in inferior performance. (2) JPOT-S outperforms JPOT-P on all metrics: this indicates that joint discriminative learning especially the compactifying effect due to Ldc to make source samples more close to their class means could help to reduce the misclassification error caused by those source samples scattered near the the decision boundary. (3) JPOT outperforms all variations. This demonstrates that the prototypical compactness loss Ldp to compactify target samples and to be more discriminative is a good strategy to boost the efficacy significantly.

5 Conclusion

This paper presents Joint Partial Optimal Transport (JPOT) for open set domain adaptation, addressing the critical challenge of negative transfer brought by unknown class samples in the target domain. We propose partial optimal transport based on global optimal transport with coupling matching matrix to exploit the most likely unknown target samples. Meanwhile, the joint discriminative feature learning helps to not only match shared class samples in both domains, but also separate samples of unknown and known classes in the target domain. This well improves the domain adaptation performance. Comprehensive experiments show that our method outperforms state-of-the-art results on various domain adaptation datasets.

Acknowledgements

This research was supported by National Key Research and Development Program of China (No. 2017YFB1400603), National Natural Science Foundation of China under grants (No. 61825205, No. 61772459, No. 61402403), and MoE Engineering Research Center of Digital Library.