MarCon: Max-Margin Contrastive Learning for Imbalanced Domain Adaptation Semantic Segmentation

1 Introduction

Semantic Segmentation (SS) is a fundamental task in computer vision that aims to assign a specific semantic label to every pixel in an image. Due to its capability for dense prediction, this technique has been extensively applied across various critical domains, including medical analysis [Ronneberger et al., 2015] and autonomous driving [Zhang et al., 2018]. However, most of the current advanced methods are built upon deep neural networks [Chen et al., 2018; Zheng et al., 2021], which are largely hungry for huge fully-supervised datasets [Long et al., 2015]. Therefore, to circumvent such labor-intensive annotation work, Unsupervised Domain Adaptation Semantic Segmentation (UDASS) [Toldo et al., 2020] provides a feasible cross-domain learning paradigm to transfer knowledge from the label-rich source domain to the vast unlabeled target domain.

∗Corresponding authors.

Typically, UDA-SS explores data distribution alignment within a domain-adaptation mechanism, which leverages the source information to facilitate precise pixel-wise classification on the target domain. Early works mostly followed the conventions of adversarial learning to enforce domain alignment from diverse aspects, e.g., input appearance, intermediate feature representations, and output spaces [Vu et al., 2019; Hoffman et al., 2018; Luo et al., 2022b]. Subsequently, self-training strategies became mainstream due to the ability to mitigate domain-shift noise via filtering or rectifying pseudo-labels [Zhang et al., 2021; Zhao et al., 2024]. Nowadays, thanks to advanced Transformer-based architectures [Hoyer et al., 2022a; Hoyer et al., 2022b] and contrastive learning techniques, UDA-SS has flourished from the perspective of representation learning [Hoyer et al., 2023].

Despite the promise, the recent UDA-SS research is still exposed to an overly idealized scenario of balanced datasets. As shown in Fig 1, SS data often reveals a severe longtailed distribution, i.e., sparse tail-class pixels are usually hidden among a large number of head-class ones, which leads the training process to be dominated by head classes, triggering the feature collapse on tail classes. Moreover, such degradation is further exacerbated in UDA-SS, thereby restricting the effective knowledge transfer from the source to the target domain. Specifically, the target tail-class pixels are overwhelmed by the noisy pseudo-labels of head classes, which in turn introduces erroneous supervision for representation learning in a vicious cycle. Therefore, UDA-SS urgently needs a tail-effective learning paradigm that can enforce strict feature margins across classes and ensure the quality of pseudo-labels simultaneously.

To address the above challenges, we propose MarCon, a method that mitigates the long-tailed problem from two synergistic aspects - representation modeling and pseudolabeling - in one coherent framework. Specifically, we first introduce Max-Margin Contrastive Learning (MCL) to rectify feature collapse. By explicitly modeling the feature space with the von Mises-Fisher (vMF) distribution, MCL reformulates pixel-level contrastive learning to enforce strict maximum margins. This constraint explicitly encourages the formation of a feature space where pixels of the same class form compact clusters, distinctly separated from other classes. Second, to guarantee the quality of the target domain pseudo-labels, we propose the Reliability-aware Filter (RaF). Leveraging the MCL-derived loss and spherical distribution affinity, RaF performs an adaptive and class-wise reliability assessment, which prevents the unreliable predictions from corrupting the model optimization. By coordinating the two modules above, MarCon explores a solution towards imbalanced UDA-SS for robust pixel segmentation. In summary, our main contributions are as follows:

• To our knowledge, this paper pioneers the exploration of a max-margin-view contrastive representation learning for imbalanced UDA-SS and proposes a tail-robust framework. As an integral part of MarCon, MaxMargin Contrastive Learning reformulates pixel-level contrastive learning, which enhances feature discriminability and mitigates the tail-class collapse.
• To realize cleaner target supervision, MarCon further integrates the Reliability-aware Filter module, which assesses target pixels' reliability and attenuates the impact of unreliable pixels to form a robust feedback loop.
• We verify the utility of our method through extensive testing on diverse UDA-SS benchmarks. MarCon

demonstrates superior adaptability, consistently improving performance across all experimental configurations.

2 RelatedWork

2.1 Unsupervised Domain Adaptation Semantic Segmentation

Unsupervised Domain Adaptation Semantic Segmentation (UDA-SS) transfers knowledge from labeled source domains to unlabeled target domains, bypassing the need for laborious pixel-wise annotations. Recent UDA-SS works typically follow a self-supervised learning paradigm [Chen et al., 2024b; Ren et al., 2024; Li et al., 2025b], especially some state-of-the-art methods adopt the teacher-student network for the target pixels pseudo-label prediction. Among them, DAFormer [Hoyer et al., 2022a] first demonstrated the potential of Transformer architectures in UDA, introducing tailored training strategies to stabilize the learning process. However, due to the high computational cost of Transformers, input images are often downsampled. HRDA [Hoyer et al., 2022b] proposed a multi-scale fusion training scheme to balance resolution with efficiency. To further enhance feature robustness, MIC [Hoyer et al., 2023] leveraged the concept of masking, forcing the model to predict masked regions to strengthen its understanding of spatial context.

2.2 Long-Tail Learning in UDA-SS

Semantic segmentation inherently suffers from long-tailed distributions, where the training process is dominated by head classes. While strategies like loss re-weighting [Wang et al., 2021] and over-sampling [Gupta et al., 2019] have proven effective in supervised settings, the domain shifts in UDASS further compound this inherent class imbalance. To mitigate this challenge in UDA-SS, various strategies have been proposed. From a data augmentation perspective, methods like [Ghiasi et al., 2021; Lee et al., 2021] alleviate tail-class scarcity by copying source-domain tail regions onto target images. Regarding pseudo-labeling, several works [Zou et al., 2018; Li et al., 2022] have been proposed to mitigate the class imbalance during the pseudo-label generation. Additionally, some approaches [Li et al., 2025a; Li and Li, 2021] introduce logit-level calibration to re-balance class predictions. Distinct from these methods, we propose a solution focusing on both the feature space and pseudo-label.

2.3 Contrastive Learning in UDA-SS

Contrastive Learning (CL) is a self-supervised learning framework that learns discriminative representations via instance similarity [Chen et al., 2020]. By minimizing the distance between positive pairs while maximizing that of negative pairs, CL learns highly discriminative feature representations. Consequently, recent advances have incorporated CL into UDA-SS to align cross-domain representations.

Some early approaches directly adapted the standard Supervised Contrastive Loss (SCL) to perform a pixel-level paradigm [Vayyat et al., 2022; Fan et al., 2024]. Subsequent works acknowledged the importance of contextual information and applied contrastive constraints at both the pixel- and the patch-level [Chen et al., 2023; Li et al.,

2023]. Besides, prototype-based methods [Jiang et al., 2022; Xie et al., 2023] contrast pixel embeddings against global semantic prototypes. Despite their efficiency, these methods overlook the inherent long-tailed distribution. In contrast, we reformulate the contrastive objective by explicitly modeling feature distributions via the vMF distribution, enforcing strict margins to secure discriminative space for tail classes.

3 Methods

In this section, we first define the UDA-SS task and then provide a detailed description of our MarCon framework.

Task Definition. In UDA-SS, the source-domain dataset Ds = {(XS n , Y S n )}Ns n=1 consists of Ns images XS n with pixellevel labels Y S n in a class space C S = {1, . . . ,C S }, and the target-domain dataset Dt = {XT n }Nt n=1 is provided without annotations and has a much broader label space C T , i.e., C S ⊆ C T . Then, the UDA-SS task aims to train a model F on training sets Ds and Dtrain t to achieve good generalization performance on the test set Dtest t . Generally, the segmentation model F is composed of a feature extractor g with a decoder h, and is ultimately trained by the cross-entropy loss.

As shown in Fig 2, our MarCon framework seamlessly integrates a well-designed contrastive learning branch into the self-training backbone. Specifically, we leverage Maxmargin Contrastive Learning (MCL) to enforce a discriminative feature space and alleviate tail-class feature collapse. Moreover, to mitigate the noisy supervision of pseudo-labels, we design a Reliability-aware Filter (RaF) to dynamically distinguish reliable pixels via MCL reliability metrics.

3.1 Self-Training Frequency-aware Framework

To alleviate domain shift and filter high-frequency noise, we integrate the following two techniques into a unified pipeline.

Teacher-Student-BasedDomainAdaption. Typically, the model F prioritizes the accurate supervision of the source training set Ds to ensure the basic semantic segmentation capability via the cross-entropy (CE) loss as:

However, relying solely on Ds to train the model F often leads to poor generalization performance on Dt due to the inherent domain shift. Consequently, we adopt the teacherstudent-based self-training paradigm, following established protocols [Hoyer et al., 2022a; Hoyer et al., 2022b]. Specifically, we employ a teacher model FΘ to generate the target pseudo-labels ˆ Y T = argmaxFΘ(XT ) for training the student model Fθ on Dtrain t via the target domain CE loss Lt :

Therefore, while using Ls to ensure strong segmentation ability, the student model Fθ also achieves effective adaptation on the target domain data via Lt . Moreover, to further alleviate the domain shift, after the teacher model FΘ generates pseudo-label ˆ Y T for the original target-domain image XT , we apply data augmentations, e.g., color jittering, Gaussian blur, and ClassMix, to generate an augmented variant X′T following DACS [Tranheden et al., 2021] for the training of the student model Fθ. Then, sharing the identical network architecture with the student model Fθ, the teacher model FΘ updates the weight Θ via the Exponential Moving Average (EMA) of the weight θ:

where α ∈ [0, 1) is the smoothing coefficient.

Frequency-Aware Feature Fusion. To capture multi-scale semantic contexts and preserve spatial details, we adopt the Transformer-based model, like SegFormer, to achieve a hierarchical encoder g, which has been proven to be applicable for semantic segmentation [Xie et al., 2021]. Considering that the drastic intra-class feature fluctuations inherent in semantic segmentation impede the process of multi-scale feature fusion, inspired by recent advances [Chen et al., 2024a; Luo et al., 2022a], we introduce a frequency-domain perspective to optimize the multi-scale feature fusion. Attributing the fluctuations to high-frequency noise, we use two complementary filters to attenuate noise-induced feature perturbations.

Specifically, we refer to this mechanism as the FrequencyAdaptive Fusion (FAF) module. First, FAF constructs a Semantic Smoothing Filter (SSF). After the encoder g extracts the feature map F, a lightweight convolutional predictor ϕ(·) with the kernel-wise softmax normalization function σ(·) derives the smoothing kernel as ksem = σ (ϕsem(F)). Besides, to compensate for the high-frequency structural information degradation caused by indiscriminate smoothing, FAF introduces a complementary Boundary Sharpening Filter (BSF) via spectral inversion. By defining δ as the discrete unit impulse kernel (i.e. an all-pass spectrum), the sharpening kernel kbnd is synthesized by subtracting the generated low-pass component from the identity: kbnd = δ - σ (ϕbnd(F)), where δ is the Kronecker delta centered at the kernel origin. Operationally, FAF applies the low-pass SSF to low-resolution features and uses the high-pass BSF to reinforce high-resolution features, thereby completing the multi-scale fusion process.

Interestingly, as shown in Fig 5a, we observe that the effectiveness of the FAF module is compromised by domain shifts, yielding only marginal improvements. To this end, by alleviating tail-class feature collapse and constructing a discriminative feature space via strict margins, our proposed maxmargin contrastive learning strategy enables the FAF module to regain its power in UDA-SS. This effect is empirically confirmed in our experimental analysis.

3.2 Max-Margin Targeted Contrastive Learning

Our primary goal is to alleviate the feature representation collapse of tail classes caused by the inherent long-tailed data distribution. To achieve this, we first theoretically quantify the discriminability of learned features.

Formally, for pixel xi with label yi, the margin γ is the cosine-similarity gap between its embedding zi and true-class centroid µyi versus the highest confidence false-class centroid µc̸=yi as:

Let Sc be the pixel set of class c, and class c's margin is γc = minx∈Sc γ(x, c). We find that the feature collapse of tail class pixels under long-tailed distributions is equivalent to their compressed class margins γ, which renders them highly susceptible to misclassification. Consequently, to facilitate robust feature learning across all classes, we aim to maximize the overall margin γC = minc γc. To this end, we conduct a theoretical analysis from the optimization perspective to identify the conditions that guarantee margin maximization within a contrastive learning framework.

Proposition 1. Let zi ∈ Z denote the ℓ2-normalized contrastive feature of pixel xi with label yi ∈ C. Under Gaussian kernel density estimation on the unit hypersphere, the optimization objective that maximizes the overall margin is:

where π(c) is the prior of class c, Z c = {zj |yj = c}, and τ is a temperature factor. A detailed proof is showed in Appendix.

To facilitate the optimization of this target, it is essential to estimate the expectation of the exponential interaction term in Eq (5). Therefore, we commence by formulating an explicit model for the feature distribution.

DistributionModeling. Given that contrastive features are constrained to the unit hypersphere, we assume they follow a mixture of vMF distributions, which is widely regarded as the directional equivalent of the Gaussian distribution on the hypersphere. Specifically, the probability density function of a unit vector z is defined as:

where µ is the mean direction (∥µ∥ = 1), κ ≥ 0 is the concentration parameter and d is the feature dimension. Cd(κ)

is the normalization constant involving the modified Bessel function of the first kind (detailed derivation is in Appendix). Max-Margin Targeted Contrastive Loss. Under the vMF assumption, the expectation term required for the kernel density estimation admits a closed-form solution:

where ˜κc = ∥κcµc +z/τ∥2 denotes the refined concentration parameter. Essentially, Eq. (7) measures the alignment between z and the class-conditional distribution, depicting how likely z is under the vMF distribution of class c. Thus, we term this quantity the Spherical Distribution Affinity (SDA), denoted as A(z, c). Substituting the closed-form solution into Eq. (5) yields the final MCL loss:

Online Parameter Estimation. To optimize the objective in Eq. (8), accurate estimation of each class's prior π(c) and distribution parameters (µc, κc) is essential. We adopt an online estimation strategy: For π(c), we maintain separate statistics for source and target domains to account for label distribution shift, i.e., πS (c) is computed from ground-truth labels, while πT (c) is updated using pseudo-labels. For vMF parameters (µc, κc), we assume a shared latent distribution across domains in the aligned contrastive space, and thus use unified centroids. Specifically, we maintain a momentum embedding ¯zc per class c, updated via the EMA operation:

where ¯z′c (t) is the mean pixel features for class c via batch-wise updating and β ∈ (0, 1) is the momentum coefficient. The centroid is the normalization form: µc = ¯zc /∥¯zc ∥2. Then we estimate the norm Rc = ∥¯zc∥2 as a sufficient statistic for κc via the numerical approximation of [Sra, 2012]:

3.3 Reliability-Boosted Self-Training

Although MCL constructs a discriminative feature space by enforcing strict margins, it cannot fully eliminate the pseudolabel noise caused by domain shift. In particular, this noise often manifests as class imbalance, causing pseudo-labels of tail classes to be overwhelmed by head classes. Therefore, as a complement to the feature space constraints, we propose the Reliability-aware Filter (RaF) to boost pseudo-label-based self-training. Specifically, inspired by the widely established small-loss criterion [Gui et al., 2021] in noisy label learning, we utilize the contrastive loss and SDA calculated for each target pixel (indexed by set Ω) as the metrics for reliability.

First, we consider pixels exhibiting lower loss are more likely to be correctly classified. Considering that a global threshold is suboptimal [Zou et al., 2018] due to the varying learning dynamics across classes and the inherent longtailed distribution, we employ a class-wise adaptive strategy. Specifically, we define a dynamic threshold ξc for the loss metric, determined by the loss value at the ρ-th percentile of pixels predicted as class c within a recent temporal window. Given that tail classes may be scarce or entirely absent within a single mini-batch, we aggregate pseudo-labeled pixels across multiple steps and compute ξc once per window to ensure more reliable statistics, especially for tail classes. Consequently, we define the loss-derived reliable subset as a subset of Ω = {i|1, . . . ,HW}, where HW quantifies the number of pixels in an image in the form of length and width, as:

where ˆ yi is the pseudo-label of pixel embedding zi. ξc is the class-adaptive threshold, determined by the loss value at the ρ-th percentile of pixels predicted as class c.

Moreover, to fully exploit potentially reliable samples, we consider pixels with high SDA as reliable ones. Therefore, we also define the affinity-derived reliable subset as Ωr-A:

where ˆ yi is the pseudo-label of pixel embedding zi and τSDA is the fixed threshold. Ultimately, RaF separates the target pixels into a reliable subset Ωr and an unreliable subset Ωu as:

To fully leverage the target pixels while mitigating potential noise, we apply differentiated supervision strategies for the identified reliable and unreliable subsets. Let qi = σ(FΘ(xi)) denote the prediction from the teacher model, and pi = σ(Fθ(xi)) denote the prediction from the student model, we formulate distinct objectives for the separated sets:

where Sharp(qi, τshp) = (qc i )1/τshp ∑Cc=1 (qc i )1/τshp for class c and τshp is the sharpening temperature. Moreover, to mitigate the inherent pseudo-label noise, the weighting factor ηn(i) computes the pixel ratio in xi's image XT n(i) whose maximum class confidence exceeds the pre-defined threshold τconf as:

Overall, by leveraging the MCL-derived metrics to identify reliable and unreliable pixels, RaF boosts pseudo-label-based supervision for segmentation training.

3.4 Overall Objective

The total objective function of MarCon is a weighted combination of segmentation losses and contrastive regularization terms across both domains. We minimize, for the source domain Ds, the supervised cross-entropy loss Lsseg and the contrastive loss LsMCL . For the target domain Dt, we use the selftraining loss Ltseg = Lr + Lu and the contrastive loss LtMCL . The overall objective is:

where λc is a trade-off factor. By jointly optimizing these losses, MarCon simultaneously achieves discriminative pixel classification and robust domain-invariant feature alignment.

4 Experiments

4.1 Experimental Setup

Datasets. Following the standard protocol [Hoyer et al., 2023; Khan et al., 2025] in UDA-SS, we evaluate our method on two widely used benchmarks: GTA → Cityscapes and SYNTHIA → Cityscapes, both of which involve adaptation from synthetic to real-world urban scenes. The source domain GTA [Richter et al., 2016] consists of 24,966 synthetic images automatically extracted from the video game Grand Theft Auto V. The SYNTHIA [Ros et al., 2016] dataset provides 9,400 synthetic images. The target domain Cityscapes [Cordts et al., 2016] contains 2,975 training and 500 validation images of real street scenes.

Implementation Details. We implement our method using the MMSegmentation framework. We adopt the same architecture as DAFormer with a MiT-B5 [Xie et al., 2021] backbone, which is pre-trained on ImageNet. The models are trained using the AdamW optimizer with a weight decay of 0.01. The initial learning rates are set to 6×10-5 for the encoder and 6 × 10-4 for the decoder. We apply a linear learning rate warm-up for the first 1.5k iterations, and linear decay afterwards. We set the teacher EMA coefficient α = 0.999 and SDA threshold τSDA = 0.968 following DAFormer. Besides, we set MCL loss weight λc = 0.4 and MCL temperature τ = 0.07 . Following the typical methods DAFormer and HRDA, we adopt random rescaling and cropping to generate input image with resolutions of 512 × 512 and 1024 × 1024, respectively. Accordingly, training is performed on one or two NVIDIA A5000 GPUs.

4.2 Experimental Results

This section presents quantitative results on GTA. → CS. and SYN. → CS., along with qualitative visualizations.

GTA → Cityscapes. As shown in Table 1, we compare our method with several current leading approaches on the GTA→Cityscapes benchmark. Specifically, we integrate our approach into multiple Transformer-based models and consistently achieve performance improvements: a gain of 2.4 mIoU with DAFormer, 1.3 mIoU with HRDA and 1.4 mIoU with MIC. Notably, our method achieves improvements over baselines on some tail classes, such as Fence and Bike. It demonstrates that our proposed MarCon effectively mitigates the bias induced by long-tailed distributions.

SYNTHIA → Cityscapes. As shown in Table 2, MarCon also demonstrates strong performance on the SYNTHIA → Cityscapes benchmark. Compared to the baseline DAFormer and HRDA, our method improves mIoU by 2.1 and 1.6, respectively. Furthermore, when integrated into MIC, MarCon yields consistent gains of 2.1 mIoU, highlighting its generalizability across different transformer-based architectures.

Qualitative results. To intuitively demonstrate the superiority of our proposed method, we visualize the segmentation maps and compare them with the strong baseline, DAFormer, on the GTA→Cityscapes benchmark, as shown in Fig 3. The baseline model struggles to handle complex scenarios, as indicated by the red dashed boxes. Specifically, in the first row, DAFormer fails to distinguish the reflective building from the sky due to appearance ambiguity. In the second row, it misses fine-grained details like the small pole and exhibits poor internal consistency on large objects (e.g., the truck). In the third row, the baseline misclassifies a building with unique textures. By integrating our method into DAFormer, we effectively alleviate these failure cases. As highlighted by the green dashed boxes, our approach successfully rectifies the misclassified regions and produces predictions that are highly consistent with the Ground Truth.

4.3 Ablation Studies

Quantitative Ablation Results. To investigate the contribution of individual modules, we conduct ablation studies on GTA → Cityscapes and SYNTHIA → Cityscapes benchmarks. As reported in Table 3, the progressive integration of our proposed components yields consistent performance improvements.The baseline achieves 68.3 and 60.9 mIoU. Applying MCL loss boosts the performance to 70.2 and 62.6, mIoU respectively. Building upon this, incorporating RaF to ensure cleaner supervision further lifts the mIoU to 70.7 and 63.0. This verifies the effectiveness of proposed modules.

Effect of MCL. To evaluate MCL's effect, we extract pixel features from a target image using baseline DAFormer, Prototype-based CL (PCL) method, and our proposed MarCon. The corresponding t-SNE visualizations are presented in Fig 4. The baseline DAFormer exhibits ambiguous decision boundaries due to its sole reliance on CE supervision, while PCL improves separation via pixel-prototype contrast. However, our MarCon achieves the most distinct boundaries and highly compact clusters. This confirms our vMF-based modeling explicitly optimizes for the maximum margin.

Unleashing the Potential of FAF with MCL. As posited in Sec 3.1, we hypothesize that MCL can amplify the efficacy of the FAF module. To verify this, we conduct a detailed ablation study in Fig. 5a. We plot the MCL+FAF (Theo.) curve (grey dashed line) to represent the hypothetical performance assuming the gains from FAF and MCL are independent and simply additive. Observing the results, the actual combined MCL+FAF (Comb.) curve (red line) outperforms this theoretical additive baseline. This margin between the red and grey curves empirically validates our claim. It demonstrates that by constructing a discriminative feature space, MCL effectively mitigates the adverse impact of domain shift on FAF, thereby amplifying the benefits yielded by FAF.

Effect of RaF. Finally, to assess the impact of RaF, we also track the mIoU of generated pseudo-labels on a target image subset across different training stages. As depicted in Fig 5b, the integration of RaF yields higher pseudo-label quality compared to the baseline by mitigating the noisy supervision of the target domain pseudo label.

5 Conclusion

In this paper, we presented MarCon, a framework designed to tackle the inherent long-tailed class distribution in Unsupervised Domain Adaptation Semantic Segmentation, thereby mitigating the consequent feature collapse and pseudo-label noise. Specifically, we focus on breaking the vicious cycle where poor tail-class representation and noisy pseudo-labels mutually degrade adaptation performance. To this end, MarCon introduces a synergistic approach combining representation modeling and reliability assessment. Specifically, our Max-Margin Contrastive Learning (MCL) reformulates the feature space using von Mises-Fisher distributions to enforce strict margins, while the Reliability-aware Filter (RaF) dynamically mitigates the impact of unreliable supervision. Extensive experiments on standard benchmarks verify that MarCon consistently outperforms current leading methods.

Acknowledgements

This paper is supported by the Ningbo Yongjiang Talent Introduction Programme (No. 2023A-399-G), CCF-NetEase ThunderFire Innovation Research Funding (NO.202516) and the Fundamental Research Funds for the Central Universities (226-2025-00065).