Robust Domain Adaptation: Representations, Weights and Inductive Bias (Extended Abstract)

1 Introduction

Deploying machine learning models in the real world often requires the ability to generalize to unseen samples i.e. samples significantly different from those seen during learning. Domain Adaptation (DA) [Quionero-Candela et al., 2009; Pan and Yang, 2009] is a well-studied approach to bridge the gap between a source and a target distributions, respectively noted pS(x, y) and pT (x, y) where x are inputs and y are labels. Unsupervised Domain Adaptation (UDA) assumes that only unlabelled data from the target domain is available during training. In this context, a natural assumption, named Covariate shift [Shimodaira, 2000; Huang et al., 2007], consists in assuming that the mapping from the inputs to the labels is conserved across domains, i.e. pT (y|x) = pS(y|x). In this context, Importance Sampling (IS) performs adaptation by weighting the contribution of sample x in the loss by w(x) = pT (x)/pS(x) [Quionero-Candela et al., 2009]. Although IS seems natural when unlabelled data from the target domain is available, the covariate shift assumption is not sufficient to guarantee successful adaptation [Ben-David et al., 2007]. Moreover, for high dimensional data [D'Amour et al., 2017] such as texts or images, the shift between pS(x) and pT (x) results from non-overlapping supports leading to unbounded weights [Johansson et al., 2019].

∗Contact Author

In this particular context, representations can help to reconcile non-overlapping supports [Ben-David et al., 2007] by learning a so-called Domain Invariant Representation [Ganin and Lempitsky, 2015];

where z := ϕ(x) for a given non-linear representation ϕ. These assume that the transferability of representations, defined as the combined error of an ideal classifier, remains low during learning. Unfortunately, this quantity involves target labels and is thus intractable. More importantly, looking for strict invariant representations (pS(z) = pT (z)) hurts the transferability of representations [Johansson et al., 2019; Liu et al., 2019; Wu et al., 2019; Zhao et al., 2019]. In particular, there is a fundamental trade-off between learning invariant representations and preserving transferability in presence of label shift (pT (y) ̸= pS(y)) [Zhao et al., 2019]. To mitigate this trade-off, some recent works suggest to relax domain invariance by weighting samples [Cao et al., 2018a; Wu et al., 2019; You et al., 2019; Cao et al., 2018b]. This strategy aligns a weighted source distribution with the target distribution

for some weights w(z). We now have two tools, w and ϕ, which need to be calibrated to obtain distribution alignment. Which one should be promoted? How weights preserve good transferability of representations?

In this paper, we show that weights allow to design an interpretable generalization bound where transferability and invariance errors are uncoupled. In addition, we discuss the role of inductive design for both the classifier and the weights in addressing the lack of labelled data in the target domain. From these theoretical insights, we derive a new learning procedure for UDA that minimizes the transferability error while controlling representation invariance with weights. We provide an empirical illustration of our framework on two DA benchmarks (Digits and Office31 datasets). We stress-test our learning scheme by modifying strongly the label distribution in the source domain. While methods based on invariant representations deteriorate considerably in this context, our procedure remains robust.

2 Preliminaries

For two random variables (X, Y ) on a given space X ×Y, we introduce two distributions: the source distribution pS(x, y) and the target distribution pT (x, y). Here, labels are one-hot encoded i.e. y ∈ [0, 1]C such that ∑c yc = 1 where C is the number of classes. We use the index notation S and T to differentiate source and target terms. We define the hypothesis class H as a subset of functions from X to Y which is the composition of a representation class Φ and a classifier class G, i.e. H = G ◦ Φ. For the ease of reading, given a classifier g ∈ G and a representation ϕ ∈ Φ, we note gϕ := g ◦ ϕ. Furthermore, in the definition z := ϕ(x), we refer indifferently to z, ϕ, Z := ϕ(X) as the representation. For two given h and h′ ∈ H and ℓ the L2 loss ℓ(y, y′ ) = ||y - y′ ||2 , the risk in domain D ∈ {S, T } is noted:

and εD(h, h′ ) := ED[ℓ(h(X), h′ (X))]. In the seminal works [Ben-David et al., 2007; Mansour et al., 2009], a theoretical limit of the target risk when using a representation ϕ has been derived:

Bound 1 (Ben David et al.) For ϕ ∈ Φ and g ∈ G

This generalization bound ensures that the target risk εT (gϕ) is bounded by the sum of the source risk εS(gϕ), the disagreement risk between two classifiers from representations dG(ϕ), and a third term, λG(ϕ), which quantifies the ability to perform well in both domains from representations. The latter is referred to as the adaptability error of representations. It is intractable in practice since it involves labels from the target distribution. Promoting distribution invariance of representations, i.e. pS(z) close to pT (z), results on a low dG(ϕ). However, it induces an unexpected trade-off when learning domain invariant representations [Johansson et al., 2019; Zhao et al., 2019]:

Proposition 1 (Invariance hurts adaptability) Let ψ be a representation which is a richer feature extractor than ϕ: G ◦ ϕ ⊂ G ◦ ψ. Then,

As a result of proposition 1, the benefit of representation invariance must be higher than the loss of adaptability, which is impossible to guarantee in practice.

3 Theory

To overcome the limitation raised in proposition 1, we expose a new bound of the target risk which embeds a new tradeoff between invariance and transferability. We show this new bound remains inconsistent with the presence of label shift and we expose the role of weights to address this problem.

3.1 ANew Trade-Off Between Invariance and Transferability

We introduce here two important tools that will guide our analysis. They are built upon F and FC; two suitable 1 classes of critics functions i.e., subset of applications from Z → [-1, 1] and Z → [-1, 1]C .

• INV(ϕ), named invariance error, that aims at capturing the difference between source and target distribution of representations, corresponding to:

• TSF(ϕ), named transferability error, that is dedicated to control if aligned representations have the same labels across domains. For that, we use our class of functions FC and we compute the IPM of Y · f(Z) (f ∈ FC and Y · f(Z) is the scalar product) between the source and the target domains:

Using INV(ϕ) and TSF(ϕ), we can provide a new bound of the target risk:

Bound 2 For ϕ ∈ Φ and g ∈ G

In contrast with bound 1 (Eq. 5), here two IPMs are involved to compare representations (INV(ϕ) and TSF(ϕ)). A new term, εT (fTϕ), reflects the level of noise when fitting labels from representations. Bounding the target risk using IPMs has two advantages. First, it allows to better control the invariance / transferability trade-off since εT (fTϕ) ≤ λG(ϕ). This is paid at the cost of 4 · INV(ϕ) ≥ dG(ϕ). Second, εT (fTϕ) is source free and indicates whether there is enough information in representations for learning the task in the target domain at first.

An interesting property of the bound, named tightness, is the case when INV(ϕ) = 0 and TSF(ϕ) = 0 simultaneously. The condition of tightness of the bound provides rich information on the properties of representations.

Proposition 2 pS(y, z) = pT (y, z) if and only if INV(ϕ) = TSF(ϕ) = 0.

3.2 ReconcilingWeights and Invariant Representations

We propose to adapt the bound by incorporating weights. More precisely, we study the effect of modifying the source distribution pS(z) to a weighted source distribution w(z)pS(z) where w is a positive function which verifies ES[w(Z)] = 1. By replacing pS(z) by w(z)pS(z) (distribution referred as w · S) in bound 2, we obtain a new bound of the target risk:

Bound 3 For ϕ ∈ Φ, g ∈ G and w : Z → R+ such that ES[w(z)] = 1:

1 See the paper for details about assumptions on critics.

As for the previous bound 2, the property of tightness, i.e. when invariance and transferability are null simultaneously, leads to interesting observations:

This proposition means that the nullity of invariance error implies distribution alignment: w(z)pS(z) = pT (z). This is of strong interest since both representations and weights are involved for achieving domain invariance. The nullity of the transferability error implies that labelling functions are conserved across domains. Furthermore, the equality ET [Y |Z] = ES[Y |Z] interestingly resonates with a promising line of work [Arjovsky et al., 2019]. Incorporating weights in the bound thus brings two benefits:

1. It raises the inconsistency issue of invariant representations in presence of label shift, as mentioned in section 2. Indeed, tightness is not conflicting with label shift.
2. TSF(w, ϕ) and INV(w, ϕ) have two disctinct roles: the former promotes domain invariance of representations while the latter controls whether aligned representations share the same labels across domains.

4 The Role of Inductive Bias

Inductive Bias refers to the set of assumptions which improves generalization, such as a specific neural network architecture or a well-suited regularization. First, we provide a theoretical analysis of the role of inductive bias for addressing the lack of labelling data in the target domain, which is the most challenging part of Unsupervised Domain Adaptation. Second, we describe the effect of weights to induce invariance property on representations.

4.1 Inductive Design of a Classifier

General Formulation.

Our strategy consists in approximating target labels error through a classifier ˜ g ∈ G. We refer to the latter as the inductive design of the classifier. Our proposition follows the intuitive idea which states that the best source classifier, gS := argming∈G εS(gϕ), is not necessarily the best target classifier i.e. gS ̸= argming∈G εT (gϕ). For instance, a wellsuited regularization in the target domain, noted ΩT (g) may improve performance, i.e. setting ˜ g := argming∈G εS(gϕ)+ λ · ΩT (g) may lead to εT (˜ gϕ) ≤ εT (gSϕ). We formalize this idea through the following definition:

Definition 1 (Inductive design of a classifier) We say that there is an inductive design of a classifier at level 0 < β ≤ 1 if for any representations ϕ, noting gS = argming∈G εS(gϕ), we can determine ˜ g such that:

We say the inductive design is β-strong when β < 1 and weak when β = 1.

In this definition, β does not depend of ϕ, which is a strong assumption, and embodies the strength of the inductive design. The closer to 1 is β, the less improvement we can expect using the inductive classifier ˜ g. We now study the impact of the inductive design of a classifier in our previous bound 3. Thus, we introduce the approximated transferability error:

leading to a bound of the target risk where transferability is target labels free:

Bound 4 (Inductive Bias and Guarantee) Let ϕ ∈ Φ and w : Z → R+ such that ES[w(z)] = 1 and a β-strong inductive classifier ˜ g and ρ := β 1-β then:

Here, the target labels are only involved in εT (fTϕ) which reflects the level of noise when fitting labels from representations. Therefore, transferability is now free of target labels. This is an important result since the difficulty of UDA lies in the lack of labelled data in the target domain. It is also interesting to note that the weaker the inductive bias (β → 1), the higher the bound and vice versa.

Predicted labels play an important role in UDA. In light of the inductive classifier, this means that ˜ g is simply set as gw·S. This is a weak inductive design (β = 1), thus, theoretical guarantee from bound 4 is not applicable. However, there is empirical evidence that showed that predicted labels help in UDA [Grandvalet and Bengio, 2005; Long et al., 2018]. A better understanding of this phenomenon is left for future work. See the paper connections between ̂ TSF(w, ϕ, ˜ g) and popular approaches of the literature.

4.2 Inductive Design of Weights

While the bounds introduced in the present work involve weights in the representation space, there is an abundant literature that builds weights in order to relax the domain invariance of representations [Cao et al., 2018a; Wu et al., 2019; You et al., 2019; Cao et al., 2018b]. In the rest of the paper we focus on weights

It is worth noting that it controls the invariance error (INV(w, ϕ) = 0). One can quantify the effect of a feature transformation ψ when designing weights;

Proposition 4 (Inductive design of w and invariance) Let ψ : Z → Z ′ such that F ◦ ψ ⊂ F and FC ◦ ψ ⊂ FC. Let w : Z ′ → R+ such that ES[w(Z ′ )] = 1 and we note Z ′ := ψ(Z). Then, INV(w, ϕ) = TSF(w, ϕ) = 0 if and only if:

while both f ϕ S = f ϕ T and f ψ S = f ψ T .

Since we do not leverage any transformation ψ, we frame it as a weak inductive design.

5 Towards Robust Domain Adaptation

5.1 Algorithm

We expose a new learning procedure which relies on weak inductive design of both weights and the classifier. This procedure focuses on the transferability error since the inductive design of weights naturally controls the invariance error. Our learning procedure is then a bi-level optimization problem, named RUDA (Robust UDA):

where λ > 0 is a trade-off parameter. ̂ TSF(w, ϕ, gw·S) and INV(w, ϕ) are computed in an adversarial manner involving two discriminators. See the paper for more details.

5.2 Experiments

Datasets. We investigate two digits datasets: MNIST and USPS transfer tasks MNIST to USPS (M→U) and USPS to MNIST (U→M). We used standard train / test split for training and evaluation. Office-31 is a dataset of images containing objects spread among 31 classes captured from different domains: Amazon, DSLR camera and a Webcam camera. DSLR and Webcam are very similar domains but images differ by their exposition and their quality. We stress-test our approach by investigating more challenging settings where the label distribution shifts strongly across domains. For the Digits dataset, we explore a wide variety of shifts by keeping only 5%, 10%, 15% and 20% of digits between 0 and 5 of the original dataset (refered as %× [0 ∼ 5]). For the Office-31 dataset, we explore the shift where the object spread in classes 16 to 31 are duplicated 5 times (refered as 5 × [16 ∼ 31]).

Comparison with the state-of-the-art. For all tasks, we report results from DANN [Ganin and Lempitsky, 2015] and CDAN [Long et al., 2018]. We report IWAN [Zhang et al., 2018], a weighted DANN where weights are learned from a second discriminator, and CDANw a weighted CDAN where weights are added in the same setting than RUDAw. A summary of results is presented in Table 1.

6 RelatedWorks

This paper makes several contributions, both in terms of theory and algorithm. Concerning theory, our bound provides a risk suitable for domain adversarial learning with weighting strategies. Existing theories for non-overlapping supports [Ben-David et al., 2007; Mansour et al., 2009] and importance sampling [Cortes et al., 2010; Quionero-Candela et al.,

2009] do not explore the role of representations neither the aspect of adversarial learning. In [Ben-David et al., 2007], analysis of representation is conducted and connections with our work is discussed in the paper. The work [Johansson et al., 2019] is close to ours and introduces a distance which measures support overlap between source and target distributions under covariate shift. Our analysis does not rely on such assumption, its range of application is broader.

Concerning algorithms, the covariate shift adaptation has been well-studied in the literature [Huang et al., 2007; Gretton et al., 2009; Sugiyama et al., 2007]. Importance sampling to address label shift has also been investigated [Storkey, 2009], notably with kernel mean matching [Zhang et al., 2013]. Recently, a scheme for estimating labels distribution ratio with consistency guarantee has been proposed [Lipton et al., 2018]. Learning domain invariant representations has also been investigated in the fold of [Ganin and Lempitsky, 2015; Long et al., 2015] and mainly differs by the metric chosen for comparing distribution of representations.

Using both weights and representations is also an active topic [Cao et al., 2018b; You et al., 2019]. Our work shares strong connections with [Combes et al., 2020] which uses consistent estimation of true labels distribution from [Lipton et al., 2018]. We suggest a very similar empirical evaluation and we also investigate the effect of weights on CDAN loss [Long et al., 2018] with a different weighting scheme since our approach computes weights in the representation space. All these works rely on an assumption at some level, e.g. Generalized Label Shift in [Combes et al., 2020], when designing weighting strategies. Our discussion on the role of inductive design of weights may provide a new theoretical support for these approaches.

7 Conclusion

The present work introduces a new bound of the target risk which unifies weights and representations in UDA. We conduct a theoretical analysis of the role of inductive bias when designing both weights and the classifier. In light of this analysis, we propose a new learning procedure which leverages two weak inductive biases, respectively on weights and the classifier. To the best of our knowledge, this procedure is original while being close to straightforward hybridization of existing methods. We illustrate its effectiveness on two benchmarks. The empirical analysis shows that weak inductive bias can make adaptation more robust even when stressed by strong label shift between source and target domains. This work leaves room for in-depth study of stronger inductive bias by providing both theoretical and empirical foundations.