Deep Reinforcement Learning Boosted Partial Domain Adaptation

1 Introduction

Deep neural networks have achieved remarkable performance on a variety of machine learning tasks. Generally, in learning theories, it is assumed that training and testing data are from the same distribution. However, data annotation for every new task can be impractical due to the prohibitive cost. Instead, it is preferable to train a model by leveraging massively available labeled data from related yet distinct domains. Therefore, domain adaptation (DA) techniques have been introduced to bridge domains of different distributions and enable knowledge transfer from label-rich source domain to label-scarce target domain.

Deep domain adaptation methods attempt to improve the learning of domain-invariant feature representations by embedding distribution matching modules into the network architectures. So far, a variety of distribution similarity measures have been incorporated into the network architecture to learn transferable representations [Long et al., 2017]. Meanwhile, a collection of adversarial learning based domain adaptation methods have been proposed, which aligns domain distributions via minimizing an approximate discrepancy in an adversarial training setting [Ganin and Lempitsky, 2014].

∗Corresponding Author

Currently, most deep domain adaptation methods assume that the source and target domains share an identical label space. Nevertheless, in real-world applications, it is often nontrivial to find source domains with the same label spaces as target domains of interest. Instead, due to the availability of large-scale labelled datasets, such as ImageNet [Russakovsky et al., 2015], a more practical yet more challenging scenario is referred to as partial domain adaptation (PDA), which relaxes the constraint of shared label spaces. It enables knowledge transfer from source domains with more classes to target domains with fewer classes without any knowledge on the size and categories of the target classes.

The PDA problems cannot be addressed by simply aligning the two domains because the knowledge migrated from irrelevant source classes can result in negative transfer. To mitigate negative interference caused by irrelevant source samples, several pioneering PDA methods have been proposed to upweight relevant source instances while down-weighting outlier source samples in domain adversarial networks [Cao et al., 2018a; Cao et al., 2018b]. In recent years, deep reinforcement learning (DRL) methods have also been implemented to learn source data selection policies. The Reinforced Transfer Network (RTNet) proposed in [Chen et al., 2020b] adopts an actor-critic algorithm [Konda and Tsitsiklis, 2000] to optimize the source data selector, while the Domain Adversarial Reinforcement Learning (DARL) framework proposed in [Chen et al., 2020a] utilizes Deep Q-Network (DQN) [Mnih et al., 2015] to learn source data selection policies.

In this paper, we propose a novel deep reinforcement learning based source data selector (DRL-DS) to boost knowledge transfer in partial domain adaptation. The DRL-based data selector is developed to automatically filter out irrelevant source samples based on their transferability. Since forcefully aligning target classes to outlier source classes can lead to negative transfer [Cao et al., 2018a], only the selected source samples will be used to train an adaptive classifier. During training, the DRL-DS model determines whether to select the source samples based on their feature representations. In the meantime, a (partial) domain adaptation model is trained to reduce the domain shift between the target and selected source samples while it is also employed to calculate the reward signals for the DRL-DS module. In this way, the DRL and DA models are jointly trained to leverage each other's capability for better knowledge transfer across domains.

Compared to conventional PDA algorithms, the integration of the DRL-DS module can lead to improved knowledge transfer from one domain to another through combining RL and DA techniques. Moreover, our DRL-based source data selector also addresses the limitations of existing DRL-based PDA algorithms. Since RTNet utilizes an on-policy DRL algorithm, it has to re-collect new samples every time the policy is updated. Besides, it also requires additional generators to calculate reconstruction error based rewards. In contrary, we employs the Dueling Double Deep Q-Network [Wang et al., 2016], which is an off-policy DRL algorithm, to learn the source data selection policies in a more efficient way. Moreover, we also design the reward functions meticulously so that better performance can be achieved without any additional network block. Although DARL also employs an off-policy DRL algorithm, it relies on domain adversarial learning to calculate rewards. In contrast, the proposed DRL-DS module is a general paradigm that can be integrated into any (partial) domain adaptation method.

The main contributions of our work are as follows:

• We have proposed a novel deep reinforcement learning based source data selector to boost knowledge transfer in partial domain adaptation. The proposed DRL-based source data selector is capable of mitigating negative transfer through eliminating irrelevant samples automatically. Moreover, it is a general paradigm that can be integrated into any (partial) domain adaptation method.
• A novel reward function is meticulously designed to better guide the learning without introducing any additional network block or relying on any network architecture.
• Extensive experiments on benchmark datasets have demonstrated the remarkable superiority of the proposed DRL-based source data selector. By adopting it, the PDA methods can outperform the state-of-the-art approaches by a large margin.

2 RelatedWork

2.1 Domain Adaptation

Domain adaptation is a branch of transfer learning which bridges domains of different distributions. The key challenge of DA is to reduce the distribution shift across different domains. Recently, DA has been combined with deep neural networks to minimize the statistical discrepancies between the deep embeddings of source and target domains. To begin with, a variety of distribution similarity measures have been incorporated into the network architecture and minimized along with the standard source classification loss for end-to-end learning of transferable representations [Long et al., 2015; Long et al., 2016; Long et al., 2017]. Distinctively, in [Xu et al., 2019], it is revealed that features with larger norm are more transferable so that the Stepwise Adaptive Feature Norm (SAFN) method has been proposed to progressively enlarge feature norms of the two domains.

Another category of DA methods introduce domain classifiers to learn transferable features in adversarial settings. Generally, the domain discriminator is to differentiate features of different domains while the feature extractor is to deceive the domain discriminator [Ganin and Lempitsky, 2014; Tzeng et al., 2017].

2.2 Partial Domain Adaptation

To address PDA problems, a number of pioneering methods have been proposed to up-weight relevant source instances while down-weighting outlier source samples [Cao et al., 2018a; Cao et al., 2018b; Zhang et al., 2018; Li et al., 2020b; Li et al., 2020a; Jing et al., 2020; Kim and Hong, 2021]. For instance, Selective Adversarial Network (SAN) [Cao et al., 2018a] introduced multiple discriminators to realize finegrained adaptation. It weighted the source instances according to their class probabilities predicted by the source classifier. In [Cao et al., 2018b], Partial Adversarial Domain Adaptation (PADA) [Cao et al., 2018b] used only one domain adversarial network and added the predicted target label probabilities to the source classifier as the class-level weights. Rather than using class-level weight, Importance Weighted Adversarial Nets (IWAN) [Zhang et al., 2018] introduced two domain classifiers so that the outputs of a second domain classifier was adopted to weight source examples. Particularly, Example Transfer Network (ETN) [Cao et al., 2019] calculated the weights of source samples according to their similarities based on a discriminative domain discriminator, and in the meantime down-weighted irrelevant source samples when updating the source classifier.

Recently, reinforcement learning algorithms have also been applied to learn source data selection policies for partial domain adaptation tasks. In [Chen et al., 2020a], Chen et al. proposed the Domain Adversarial Reinforcement Learning (DARL) method to learn data selection policies automatically through DQN. The calculation of their reward signals relied on the domain adversarial learning framework. In [Chen et al., 2020b], Chen et al. introduced the Reinforced transfer network (RTNet) to eliminate outlier source classes through a reinforced data selector while combining both high-level and pixel-level information. These two pioneering DRL-based PDA approaches have validated the feasibility of applying RL in the context of PDA. In this work, we propose a novel DRL-based source data selector to boost knowledge transfer in partial domain adaptation and adopt a Dueling Double DQNmodel to verify its feasibility.

3 Method

3.1 Problem Statement

The partial domain adaptation scenario constitutes source domain Ds = {(xsi , ys i )}ns i=1 which has ns labeled examples associated with |Cs| classes, and the target domain Dt = {xtj }nt j=1 which has nt unlabelled examples associated with

|Ct| classes. The source domain subsumes the target domain so that Ct ⊂ Cs. Since the two domains are drawn from different probability distributions p and q, p ̸= q and pCt ̸= q, where pCt represents the distribution of source data in the target label space. Direct alignment of these two distributions can result in negative transfer because pCs\Ct and q are not overlapped, where pCs\Ct indicates the distribution of source instances in the outlier categories. Therefore, it is important to eliminate irrelevant source samples in PDA tasks while reducing the distribution shift between pCt and q.

The objective of this paper is to develop a DRL-based source data selector (DRL-DS) to boost knowledge transfer in PDA. As depicted in Fig. 1, the proposed DRL-DS module is a general paradigm that can be integrated into any (partial) domain adaptation method. In this paper, it is integrated into a domain adaptation method (SAFN) [Xu et al., 2019] and a partial domain adaptation method (ETN) [Cao et al., 2019] respectively to fully prove its effectiveness and generalization capability. The objective of the DRL-DS module is to learn data selection policies automatically so that negative transfer can be mitigated through eliminating outlier source samples. In the meantime, the (partial) domain adaptation module aims to learn domain-invariant feature representations.

3.2 DRL-DSModule

In this paper, the source data selection task is modeled as a Markov Decision Process (MDP) defined by a tuple M = (S,A,R, P, γ), where S, A, R, P, γ denote the state space, action space, immediate reward, state transition function, and discount factor, respectively. At time step t, an action at ∈A is executed based on the current state st ∈S. The DRL agent then receives a reward R(st, at), and transits to a new state st+1. A policy π(a|s) which specifies the mapping from a state s to an action a is assessed by the Q-value function defined as:

The goal is then to maximize the expectation of cumulative reward, which can be solved by the Q-learning algorithm shown in the following:

Thereby, the optimal policy can be derived as π∗ (s) = argmax Q∗ (s, a).

a

In this paper, we employ the Dueling Double Deep QNetwork to figure out the optimal data selection policies and verify the feasibility of the proposed DRL-DS based PDA framework. Given a batch of source data {xsi }nb i=1 = Xsb , where nb and b denote the batch size and ID respectively, the feature representations of these source samples F (Xsb ) can be obtained, where F indicates the feature extractor in the (partial) domain adaptation module. The DRL-DS module then maps F(Xsb ) to a series of actions {asi }nb i=1 = Asb , which in turn weights the source instances while updating the parameters of the (partial) domain adaptation network. Meanwhile, the rewards of the actions {Rs i }nb i=1 are determined according to the transferability of the source samples. In the following, the state, action, reward, and training of our DRL-DS module are introduced in detail.

State. Feed a batch of nb source samples to the feature extractor of the (partial) domain adaptation module, a number of nb features Ft(Xsb ) = [Ft(xs1 ), · · · , Ft(xsnb )] can be generated, where Ft(xsi ) represents the feature vector extracted from source instance xsi at time step t. In the DRL-DS module, a state is defined as one source feature vector Ft(xsi ) and at each state, an action is taken to determine whether to keep or discard the corresponding source sample. In this way, the weights of the nb source instances can be derived via passing the nb source feature vectors to the DRL-DS module successively. The (partial) domain adaptation network is then optimized based on the selected source samples so that a number of nb new features Ft+1(Xsb ) can be generated by the updated feature extractor. For a specific source sample xsi , if it is selected at time step t while the episode is not completed, its next state will be Ft+1(xsi ). Otherwise, a terminate state will be triggered. Therefore, each batch of source samples are trained T times where T represent the episode length and at each training step, it can generate nb experiences.

Action. In our DRL-DS module, the action space is binary and each action asi ∈ {0, 1} denotes whether a source sample is retained or discarded. Specifically, asi = 1 means to keep source sample xsi while asi = 0 means to eliminate it. The output of the DRL Network is a two-dimensional Q-value vector mapped from the input state through fully connected layers. The optimal action at state Ft(xsi ) is then determined as:

Reward. The reward function is shaped to provide feedback signals and thereby guide the learning of source data selection policies. As the objective is to select source samples that are more relevant to the target domain, we evaluate the transferability of source instances by measuring their similarity to the target instances and design the reward function based on this transferability. At each training step, in addition to source data, a batch of nb target samples {xtj }nb j=1 = Xtb are also fed into the (partial) domain adaptation module. Similarly, nb feature vectors of these target data Ft(Xtb ) = [Ft(xt1 ), · · · , Ft(xtnb )] can be obtained through the feature extractor. The relevance of each source instance to the target domain can then be estimated by measuring the similarity between its feature vector and those of the target batch. To begin with, the transferability of each source sample to the target domain is evaluated through calculating two functions, i.e.,

and

(a) DRL-DS + SAFN

(b) DRL-DS + ETN

Figure 1: DRL-based data selector for partial domain adaptation. The DRL-DS module eliminates irrelevant source samples based on the estimated Q-values while the (partial) domain adaptation module learns domain-invariant feature representations using the target and selected source data. The reward signals are determined based on the relevance of the source samples to the target domain.

The first function is based on the cosine distance. It is a measure of dissimilarity between two vectors of an inner product space and is invariant to scaling. The second function is based on the Canberra distance. It is a sensitive metric which measures fraction differences between point pairs in a vector space. The smaller the distance values, the greater the match between the feature vectors. Hence, the relevance of each source sample is determined considering both angle and distance between its feature vector and those of the target batch. Based on these two measures, the reward of taking an action asi is designed as:

where ⊕ is the exclusive-or operation, A = (asi == 1),

boolean functions, and λ1, λ2, λ3, λ4 denote four constants to determine the range of the reward. For each measure, a positive value will be obtained if a source is selected while it exhibits higher relevance to the target domain. Similarly, a positive value will also be obtained if a source is eliminated while it exhibits lower relevance to the target domain. Otherwise, a negative value will be triggered. In our DRL-DS module, the reward signals are bounded between -1 and 1 to provide explicit guidance to the agent so that it can learn to distinguish good actions from bad ones efficiently.

Training. Our DRL-DS module maintains two deep neworks, i.e., an online network with parameters θ and a separate target network with parameters θ-. During training, the online network is updated constantly while the target network is softly updated via polyak averaging to stabilize the training. Instead of using the ϵ-greedy strategy for exploration, our model implements NoisyNets [Fortunato et al., 2017] rather than conventional fully connected layers to achieve more efficient exploration in a consistent way. A linear layer can be expressed as y = ωx + b, where x, y, ω and b denote the input, output, weight matrix and bias, respectively. To achieve exploration, the uncertainty can be added by factorized Gaussian noises so that the weight matrix can be re-formatted as:

and the corresponding bias can be re-written as:

where f(ϵ) = sgn(ϵ)√|ϵ|, µωi,j , σω i,j , µbj , σb j are network parameters, and ϵi, ϵj are random noises. In this way, the state value V (st) and the advantage A(st, at) are estimated through noisy fully connected layers and thereafter combined to produce the Q-value using:

where θC, θV , θA represent the parameters of the common network, the state value stream and the advantage stream, respectively. At each training step, nb new transitions are stored in an experience replay buffer, and a random mini-batch of the stored transitions is sampled from the buffer. The Q-value is updated by minimizing the Huber loss between the estimated Q-value Q(st, at; θ) and the target Q-value which is defined as:

where γ is the discount factor and β is a binary value which equals zero if the episode terminates at step t+1 or one otherwise. It is worth mentioning that the proposed DRL-DS model can be directly integrated with any DA method without modifying either the DA or the DRL-DS model. This is because the DRL-DS model regards source features as its inputs and calculates rewards based on source and target features, while feature extractor is general in all the DA models.

3.3 Domain AdaptationModule

In this paper, the proposed DRL-DS module is integrated with both a domain adaptation method (SAFN) and a partial domain adaptation method (ETN) respectively to evaluate its generalization capability.

SAFN. Stepwise Adaptive Feature Norm (SAFN) learns a feature extractor F with parameters θf and a classifier C with parameters θy. The parameters of C are updated by minimizing the softmax cross entropy loss on the selected source samples. Meanwhile, in addition to the cross entropy loss, the parameters of F are also updated by enlarging the feature norms of both the target and the selected source samples. Hence, SAFN is dedicated to solve the following optimization problem:

where ys i is the one-hot label for xsi , w(xsi ) ∈ {0, 1} is the binary weight of xsi determined by the output action asi of the DRL-DS module, h(x) = (‖·‖2 ◦F)(x), θ0 is the current parameters of F , ∆r represents a positive residual scalar to control the norm enlargement, Ld(·, ·) denotes the L2-distance, and λ is a hyperparameter to trade off the objectives.

ETN. Example Transfer Network (ETN) jointly learns a domain-invariant classifier and a progressive weighting scheme which quantifies the transferability of source samples. It learns a feature extractor F with parameters θf , a classifier C with parameters θy and a discriminator D with parameters θd. In addition, it also learns an auxiliary domain discriminator ˜D to quantify source samples' transferability. Combined with the DRL-DS module, the domain adaptation network is updated through min θf ,θyEC-ED and min θd ED, where

In Eq. 12, λ is a trade-off parameter, w(xsi ) denotes the binary weight generated by the DRL-DS module and H(·) is the entropy loss. In addition, W(xsi ) is the weight generated using the auxiliary discriminator of the ETN model so that:

Generally, the value of ˜D(F(xti )) will be smaller if a source example is more probable to be in the shared label space. Since ˜D(F(xti )) is closer to one, the weights ˜D(F(xti )) in each batch size are normalized as W(xsi ) = W(xsi ) 1 nb nb∑ i=1 W(xsi ) .

4 Experiments

4.1 Setup

The proposed method is evaluated on three datasets.

Office-31 [Saenko et al., 2010] is a widely used domain adaptation dataset containing 4,652 images in 31 categories from three domains, i.e., Amazon (A), Webcam (W) and DSLR (D). Following [Cao et al., 2018a], the same ten categories are selected as target domains to create six PDA tasks.

Office-Home [Venkateswara et al., 2017] is a more challenging dataset that contains about 15,500 images in 65 categories from four different domains, i.e., Artistic (Ar), Clipart (Cl), Product (Pr), Real-World (Rw). Following the settings in [Cao et al., 2018b], the first 25 categories in alphabetic order are selected as target domains to create twelve PDA tasks.

VisDA2017 [Peng et al., 2017] is a large-scale dataset which includes over 280,000 images across 12 categories and aims to bridge the significant gap between synthetic and real domains. Following the settings in [Cao et al., 2018b], we select the first six categories in alphabetic order as target categories to create the Synthetic-12→Real-6 task.

The DRL-DS module consists of one common fully connected layer, two noisy fully connected layers for the state value stream and two noisy fully connected layers for the advantage stream. During training, the backbone network, ResNet-50, is pre-trained on ImageNet while the other layers are trained from scratch. The DA module is trained via SGD with a batch size of 32 and a learning rate of 1e-3. The DRL-DS module is trained using Adam with a batch size of 32 and a learning rate of 1e-4. The episode length is set to five. The discount factor in Eq. 10 is set to 0.9 and λ1, λ2, λ3 and λ4, in Eq. 6 are set to 0.2, 0.5, 1.8 and 0.5, respectively. We compare the proposed method with a variety of state-ofthe-art methods. For all the baseline methods, we either refer to the reported results in [Cao et al., 2019; Li et al., 2020b; Chen et al., 2020b; Chen et al., 2020a; Xu et al., 2019; Li et al., 2020a; Jing et al., 2020; Kim and Hong, 2021] or calculate the average values of three runs using the original code. The proposed methods are also trained three times to calculate the average values.

4.2 Results and Discussions

The classification results on the six tasks of Office-31 are shown in Table 1. It is observed that the two proposed methods significantly outperform most of the baseline methods with average accuracies 97.21% and 98.41%, respectively. Compared to SAFN and ETN, DRL-DS + SAFN and DRLDS + ETN achieve a 4.56% and a 4.18% improvement in average accuracy, respectively. It is worth mentioning that the DRL-DS + ETN method achieves state-of-the-art accuracy. By integrating the proposed DRL-DS module, the performance of the (partial) domain adaptation methods are consistently improved on all tasks by a large margin with similar convergence speed, which convincingly demonstrates the significance of the proposed DRL based source data selector.

In addition, it is noticed that DAN, DANN, ADDA and RTN all lead to performance degradation compared to ResNet. This is because direct aligning domains with different label spaces can result in negative transfer. In contrast, the PDA methods achieve better performance on most tasks since their weighting schemes mitigate negative influence caused by outlier source data. Therefore, it is crucial to eliminate irrelevant source instances while aligning the two domains.

Different fromOffice-31, Office-Home and VisDa2017 are larger and more challenging. The classification results on these two datasets are shown in Table 2 and 3, respectively. Similarly, DRL-DS + SAFN and DRL-DS + ETN outperform SAFN and ETN consistently on all tasks, respectively. By incorporating the proposed DRL based source data selector, DRL-DS + SAFN and DRL-DS + ETN gain 1.26% and 3.72% improvement on the Office-Home tasks, respectively, and gain 3.95% and 17.17% on the VisDa2017 task, respectively. Therefore, it proves that the proposed DRL-DS module is superior to filter out irrelevant source data automatically and thereby boost knowledge transfer in PDA tasks. Compared to existing RL-based PDA methods, our method is capable of achieving better performance in an off-policy manner without relying on any network architecture or requiring any additional network block for reward calculation.

In addition, the fraction of shared classes in the selected and eliminated source samples are depicted in Fig. 2(a). Theoretically, source data in shared classes should be selected while those in unshared classes should be eliminated during training. It is observed that the source samples selected by the DRL-DS module mostly belong to the categories shared between the source and target domains, while the instances eliminated by the DRL-DS module rarely belong to the shared classes. This demonstrates the capability of the proposed DRL-based data selector to boost knowledge transfer through filtering out irrelevant source instances. Lastly, the t-SNE embeddings with domain information are demonstrated in Fig. 2. The source and target representations are in blue and red respectively. Before training, the source and target correlated features are not well aligned and feature representations collide into a mess. After training, DRLDS + SAFN and DRL-DS +ETN are capable of aligning the target samples to corresponding source domain clusters and discriminating different classes in both domains, which manifests the effectiveness of DRL-DS to boost PDA performance.

Despite the superiority of DRL-DS, the key limitation of the proposed method is its sensitivity to hyper-parameters. Specifically, the episode length of RL and the batch size of DA are two important parameters. If the episode length is too small, the RL task cannot be well formulated, and if it is too large, the process can be redundant. For instance, DRL-DS + SAFN achieves an average accuracy of 96.61, 93.33 and 94.58 on the a→w task with the episode length being set to

5, 2 and 10, respectively. Similarly, since the transferability of source samples is only calculated based on a batch of data, batch size can be a sensitive hyper-parameter, especially on larger dataset. If the batch size is too small, the similarity cannot be approximated accurately and stably, while it also cannot be too large due to limited computational resources and increased computational time. For instance, DRL-DS + SAFN achieves an average accuracy of 96.61, 95.25 and 86.67 on the a→w task with the batch size being set to 32, 8 and 4, respectively. Though these hyper-parameters can be obtained empirically, we still aim to deal with this limitation by redesigning the reward or the framework in the future.

5 Conclusion

In this paper, we propose a DRL-based source data selector to boost cross-domain knowledge transfer in PDA tasks. The developed DRL-based data selector is a general paradigm that enables automatic and efficient elimination of irrelevant source instances to circumvent negative transfer and in turn, boost positive transfer across domains. Experimental results on different benchmark datasets have demonstrated the significance of the proposed DRL-based data selector.

Acknowledgements

This research is supported by the Agency for Science, Technology and Research (A*STAR) under its AME Programmatic Funds (Grant No. A20H6b0151) and Career Development Award (Grant No. C210112046).