BAMFair: Barycenter Aligned Mediation for Fairness Across Multiple Sensitive Attributes

1 Introduction

In recent years, machine learning has been increasingly applied to assist decision-making in critical social scenarios including visual recognition [Xie et al., 2026c; Xie et al., 2026a], recommendation systems [Xie et al., 2026b; Liu et al., 2026; Liu et al., 2025b], and justice [Mehrabi et al., 2021], etc. With the rapid advancement of machine learning, concerns about its fairness have intensified [Zhang and Harman, 2021]. In practice, machine learning algorithms often exhibit unfair behaviors related to sensitive attributes such as gender, race, age, and region. For instance, Amazon's recruitment model exhibited gender bias [Smith Brain Trust, 2018], favoring male candidates; in fairness testing via the FairLLM benchmark [Zhang et al., 2023], ChatGPT was found to have fairness flaws across eight sensitive attributes in movie recommendations.

Therefore, algorithmic fairness has become increasingly important to reduce unintentional biases in machine learning. Existing research has proposed various fairness algorithms [Smith Brain Trust, 2018]. Recently, main approach is first to divide data into different subgroups based on predefined sensitive attributes, and then to improve fairness by addressing the imbalance among different subgroups. For instance, Reweighting (RW) [Kamiran and Calders, 2011] assigns different weights to different subgroups to improve fairness; Fair-SMOTE [Chakraborty et al., 2021] achieves fairness by creating synthetic data for minority subgroups to fix the imbalance. Currently, most existing work on fairness focuses on debiasing for a single sensitive attribute. However, it is common for multiple sensitive attributes occurring simultaneously in real-world applications. Consider an example of a loan approval scenario with two sensitive attributes (e.g., gender and race), as shown in Figure 1. We observe that when considering only gender, the approval rate for black people is significantly lower. This will not only exacerbate gender inequality, but also may undermine public trust in the fairness of the financial system [Smith Brain Trust, 2018]. Only by considering both race and gender can we ensure that different subgroups have comparatively fair opportunities in loan approval. Notably, other sensitive attributes (e.g., marital status and educational level) can also influence decision outcomes in practice. Ignoring such attributes may cause severe discrimination in the real world. Thus, it is critical to address fairness across multiple sensitive attributes.

However, existing methods primarily focus on single sensitive attribute and lack exploration of multiple sensitive attributes. Therefore, the first challenge is (CH1): Fairness models based on sensitive attribute grouping cannot remain effective when extended to multiple (two or more) sensitive attributes. To verify the importance of multiple sensitive attributes, we conducted a preliminary experiment comparing scenarios with different sensitive attributes, as shown in Table 1. When we consider only gender, the fairness of gender improves. However, racial fairness becomes even worse than the baseline with no sensitive attributes. Only by considering both gender and race can the model achieve fairness for both groups. Although some studies have explored fairness with two sensitive attributes, they remain difficult to extend to scenarios with more sensitive attributes. For instance, although Fair-SMOTE can be extended to more sensitive attributes, it requires C 2 2n alignment operations between 2n subgroups (formed by pairwise combinations of sensitive attributes) in a binary classification task with n sensitive attributes, which leads to a high computational cost.

Furthermore, as the number of sensitive attributes increases, the number of subgroups divided by these attributes grows correspondingly. This makes fairness optimization across subgroups more challenging. For instance, when fairness objectives expand from singular focus on racial fairness to multiple targets encompassing race, gender, and age, conflicts emerge in multi-objective optimization. We thus pose the following challenge (CH2): The existing work overlooks to negotiate an agreement among inconsistent gradient up-

1MEOP is a fairness metric, where smaller values indicate better fairness. Its formal definitions is provided in Section 3.2.

dates between accuracy and complex fairness objectives.

In this paper, we propose a novel fairness framework named BAMFair, a Barycenter Aligned Mediation framework for Fairness across Multiple Sensitive Attributes. To address CH1, BAMFair introduces a global fair barycenter and innovatively transforms the fairness problem into minimizing the distances from individual subgroups to this barycenter. This approach effectively avoids the high complexity of na¨ıve methods that rely on pairwise alignments between all subgroups. By focusing on subgroup-barycenter matching instead, this design significantly reduces the complexity of fairness optimization. We provide a detailed time complexity analysis in Section 4.1. Furthermore, by reducing the number of optimization objectives, the proposed alignment strategy notably alleviates CH2. Specifically, GBA changes the number of fairness goals from O(T 2 ) pairwise matches to only O(T) alignments, where T is the number of subgroups. By significantly reducing the total number of optimization targets, our method makes it easier to negotiate an agreement among inconsistent gradient updates between accuracy and fairness objectives. To further mitigate CH2, BAMFair incorporates a Nash Fairness Mediator (NFM), which dynamically ensures the accuracy and fairness of the model.

We summarize our main contributions as follows:

• We propose a novel framework BAMFair, which realizes the subgroup-barycenter alignment strategy, thus effectively addressing the fairness problem with multiple sensitive attributes.
• The BAMFair framework contains two modules: GBA and NFM. Specifically, GBA innovatively transforms the fairness problem into minimizing the distances from individual subgroups to the barycenter. NFM negotiates an agreement among inconsistent gradient updates between accuracy and fairness objectives.
• Extensive experiments conducted on four real-world datasets demonstrate the effectiveness and scalability of our proposed BAMFair, compared with other state-ofthe-art methods.

2 RelatedWork

With the widespread deployment of machine learning, fairness models have gained significant attention. Fairness methods are categorized by their mitigation stage into pre-processing, in-processing, and post-processing. Preprocessing debiases data before training [d'Alessandro et al., 2017], in-processing embeds constraints during training [Dai and Wang, 2021], and post-processing adjusts model predictions [Dai et al., 2024]. While flexible and model-agnostic, the effectiveness of pre-processing and post-processing is limited because the training process itself can introduce new biases [Zhang et al., 2025]. Consequently, many methods focus on adjusting the training process. For instance, [Kamishima et al., 2012] developed a bias remover using discrimination-aware regularization, while [Lokhande et al., 2020; Liu et al., 2019] incorporate fairness metrics directly into loss function.

Despite these advancements, research on multiple sensitive attributes remains limited [Chen et al., 2024]. Although some existing methods such as [Chakraborty et al., 2021] and [Peng et al., 2022] can handle two sensitive attributes, they suffer from high computational complexity and inconsistent gradient between accuracy and complex fairness objectives. A na¨ıve solution to extend current methods to multiple sensitive attributes is pairwise alignment between subgroups, which also leads to a large number of targets and heavy computational costs. In this paper, we propose BAMFair, an inprocessing method designed for efficient and scalable fairness optimization involving multiple sensitive attributes.

3 Preliminary

3.1 Notations

Consider a binary classification task with dataset D = {(x, y, s)}, where x ∈ Rd is the input attribution, y ∈ {0, 1} is the binary ground truth label, and s = (s1, s2, . . . , sk) ∈ {0, 1} represent k binary sensitive attributes. We use ˆ y ∈ {0, 1} to represent the predicted label. Let G = (g1, g2, . . . , gT ) represent the set of subgroups divided by sensitive attributes, where T = 2k .

3.2 Fairness EvaluationMetrics

We define two fairness metrics for a binary label y ∈ {0, 1}. To handle multiple sensitive attributes, we evaluate the disparity between any two subgroups gr, gw ∈ G.

Definition 1. Demographic Parity (DP) [Dwork et al., 2012]. DP requires the model's predictions to be independent of sensitive attributes. The disparity is quantified by MDP, which measures the maximum difference in acceptance rates between any two subgroups:

where P(·) denotes the probability measure.

Definition 2. Equal Opportunity (EOP) [Hardt et al., 2016]. EOP requires that the true positive rate be consistent across all subgroups. The deviation is measured by MEOP, reflecting the maximum gap in positive prediction probabilities for individuals with positive ground-truth labels:

4 Method

In this section, we will introduce the details of our proposed method BAMFair for solving CH1 and CH2. The framework of BAMFair is shown in Figure 2. BAMFair mainly has two modules, i.e., global barycentric alignment (GBA) module and nash fairness mediator (NFM) module. Specifically, BAMFair first obtains embeddings of each subgroup through the backbone network ψ [Liao et al., 2025; Liu et al., 2025a; Xie et al., 2021]. Then, GBA constructs a fair barycenter distribution of all subgroups, and continues to calculate the distance from each subgroup distribution to the barycenter to construct the fairness loss Lfair. Finally, NFM dynamically balances different losses, outputting the optimal parameter update direction.

4.1 Global Barycentric Alignment (GBA)

We first introduce the Global Barycentric Alignment (GBA) module in BAMFair, which consists of two key steps: global barycentric construction and fairness loss construction. The global barycentric construction aims to calculate distribution of global fairness barycenter through subgroups. The fairness loss construction aims to quantify the distance between each subgroup and barycenter to form the fairness loss. These two steps align subgroups with the barycenter, thereby enhancing fairness with limited time cost.

Step 1: Global Fair Barycenter Construction

To quantify the prediction distribution of each subgroup, we first discretize the model's output scores into histograms. For subgroup gt ∈ G, the range of prediction scores is divided into M continuous non-overlapping bins. For the i-th bin of gt, vt,i represents the center value of the i-th bin in gt, and at,i represents the proportion of samples in gt with scores falling into the i-th bin, satisfying at,i ≥ 0 and ∑Mi=1 at,i = 1. The discretized distribution of gt is thus Pt = {(at,i, vt,i)}Mi=1 . After that, we construct a global fair barycenter P ∗ as an unbiased reference distribution, which is also represented as a histogram with N bins, i.e., P ∗ = {(bj, µj)} (for j = 1 to N), where µj represents the center value of the j-th bin of the barycenter, while bj represents the probability mass of the jth bin of the barycenter, satisfying bj ≥ 0 and ∑Nj=1 bj = 1. We define πt,ij as the transport plan, representing the mass transferred from the i-th bin of subgroup t to the j-th bin of the barycenter, Then, the optimal transport problem [Wang et al., 2026a] from subgroups to barycenter is formulated as:

where the distance matrix Ct,ij = ∥vt,i - µj∥22 , and H(π) = - ∑i,j πt,ij(log πt,ij -1) is the entropy of the transport plan, and ε controls the weight of entropy.

We define potential functions ft,i, qj. The transport plan πt,ij can be expressed as:

(a) Na¨ıve Alignment Method

(b) Framework of BAMFair

Figure 2: The subfigure (b) is the overview of BAMFair, including two modules: (1) GBA constructs a global barycenter from subgroups and only requires T = 6 alignments. In contrast, the subfigure (a) shows that na¨ıve pairwise alignment method requires C2T = 15 operations for T = 6 subgroups. (2) NFM negotiates an agreement among inconsistent gradient updates between different objectives, aiming to handle conflicting requirements and achieve the optimal update direction.

Meanwhile, by using constraints △ imposed on the marginals, we derive:

After updating transport plan, we can fix πt,ij and update µj :

Upon convergence, we obtain the histogram distribution P ∗ = {bj , µj} of the global barycenter.

Step 2: Fairness Loss Construction

After getting the barycenter distribution P ∗ , the fairness loss construction continues to quantify the deviation between each subgroup distribution Pt = {at,i, vt,i} and the barycenter P ∗ by calculating the Sinkhorn distance [Wang et al., 2024; Wang et al., 2025a]:

Finally, the fairness loss Lfair is defined as the average of Sinkhorn distances from all subgroups to the barycenter:

According to the triangle inequality property, the distance between any two arbitrary subgroups gr and gw is upper-bounded by their distances to the global barycenter: S(Pr, Pw) ≤ S(Pr, P ∗ ) + S(Pw, P ∗ ). Consequently, minimizing Lfair in Eq.(9) effectively constrains the pairwise disparity across all subgroup combinations. This strategy resolves CH1 by reducing the complexity from O(T 2 ) pairwise comparisons to O(T) alignments, providing a scalable fairness guarantee for multiple sensitive attributes.

Complexity Analysis

Aligning subgroups with a global fair barycenter can reduce the computational burden while ensuring fairness across multiple sensitive attributes. For all subgroups, the time complexity of fairness alignment is O(T · M · N), compared to traditional pairwise alignment with time complexity O(T 2 · M 2 ). This ensures BAMFair maintains efficiency while enhancing fairness for multiple sensitive attributes.

4.2 Nash Fairness Mediator (NFM)

In addition to the fairness loss Lfair, we introduce an accuracy loss Lacc and a contrastive learning loss Lcon. The accuracy loss quantifies the discrepancy between the model's predictions and the ground-truth labels, ensuring the model retains predictive power while optimizing for fairness. For binary classification tasks, it is defined using cross-entropy loss:

where y ∈ {0, 1} is the ground-truth label, ˆ y ∈ [0, 1] is the model's predicted probability of positive class, and the expectation is taken over the dataset. The contrastive learning loss Lcon enhances the model's ability to represent features irrelevant to sensitive attributes, reducing reliance on sensitive attributes and improving robustness and fairness of features[Liao et al., 2024b; Liao et al., 2024a]:

where hm and hn form a positive pair, with hs as negative samples. sim(·, ·) uses cosine similarity to measure feature vector similarity, and the temperature parameter τ regulates the sharpness of the similarity distribution. Positive sample pairs consist of samples that differ in sensitive attributes yet exhibit high similarity in non-sensitive attributes and share consistent labels. This design compels the model to learn the common, discriminative features across such pairs, thereby reducing its over-reliance on sensitive attributes for prediction and enhancing fairness.

Inspired by [Navon et al., 2022], the nash fairness mediator module aims to dynamically balance conflicts in multi-objective optimization, specifically coordinating gradient conflicts between the accuracy loss Lacc, fairness loss Lfair, and contrastive learning loss Lcon, ensuring the model does not sacrifice accuracy while improving fairness[Wang et al., 2025b; Wang et al., 2026b]. First, compute the gradients of each loss: ∇Lacc (accuracy gradient), ∇Lfair (fairness gradient), and ∇Lcon (contrastive learning gradient), and construct the gradient matrix G = [∇Lacc,∇Lfair,∇Lcon], denoted as E = [e1, e2, e3].

The Nash bargaining solution aims to find non-negative weights α = (α1, α2, α3) that satisfy the Nash equilibrium condition through a weighted gradient combination:

where d is the parameter update direction, and ·⊤ denotes the transpose operation. This condition ensures equal marginal gains for each loss, preventing a single loss from dominating the update direction due to scale advantages.

α(0) is initialized to a uniform distribution, with αk ≥ 0 and ∑k αk = 1. In each iteration, update the weights α by solving the following optimization problem:

This objective function is derived from the Nash bargaining solution, where e⊤k d(t) represents the projection of the gradient ek onto the current update direction d(t) , reflecting the contribution of the loss k to the current update. The constraints ensure that the weights are non-negative and sum to 1, maintaining the property of a convex combination. To solve this optimization problem efficiently, we can use Lagrange multipliers. Construct the Lagrangian function:

The final solution for the weights can be expressed as:

This formula demonstrates that the weight of each loss is inversely proportional to its projection onto the current update direction, balancing the contributions of different gradients.

After obtaining the new weights α(t+1) , update the parameter update direction:

This direction optimizes all losses simultaneously, avoiding bias caused by gradient scale differences.

4.3 Putting Together

Model parameters θ are updated using the coordinated direction d from NFM, with a learning rate η:

By operating on gradients (which encode local optimization directions) rather than raw loss values, the model avoids bias from differing loss magnitudes.

5 Experiment

In this section, we conduct extensive experiments to address five research questions: RQ1: How does BAMFair perform compared to the state-of-the-art fairness method? RQ2: What are the effects of different model components? RQ3: How does our method perform in terms of accuracy, fairness and time cost when it is extended to multiple sensitive attributes? RQ4: Why do we choose NFM over other multiobjective optimization methods?

5.1 Experimental Setup

Datasets

We conduct research using four real-world datasets: Adult [Becker and Kohavi, 1996], Compas [ProPublica, 2016], Credit [Yeh, 2009], and Mep16 [AHRQ, 2016]. For experiments involving two sensitive attributes, we select gender and race from the datasets; for three sensitive attributes, we add age as an additional sensitive attribute. For each experiment trial, we split the data into 70% training, 10% validation, and 20% testing sets. We repeat this procedure 10 times for statistical analysis and report the mean and standard deviation.

Comparison Methods

We compare BAMFair with several state-of-the-art fairness improvement methods, covering pre-processing, inprocessing, and post-processing methods. Pre-processing includes RW [Kamiran and Calders, 2011], Fair-SMOTE [Chakraborty et al., 2021]. In-processing includes MAAT [Chen et al., 2022], FairGP [Luo et al., 2025]. Postprocessing includes ROC [Kamiran et al., 2012], FairMask [Peng et al., 2022], SFM [Hu et al., 2024]. We also compare our method with a traditional model without fairness restriction named Default [Chen et al., 2024].

Evaluation Metrics

The evaluation metrics for the model include Accuracy (Acc) 2 , MDP , and MEOP . Smaller values of the fairness metrics MDP andMEOP indicate the model is fairer.

2 In this paper, we use the percentage of correctly predicted samples as Acc.

Implementation Details

For binary classification tasks, we use a logistic regression model to generate classification results. We set the batch size as 128 and adopt Adam optimizer with the learning rate η as 0.002. We set M (number of bins for subgroup distributions) to 100. The temperature parameter τ is set to 0.07. ε (entropy weight in optimal transport) is set to a conventional value of 0.1 [Cuturi, 2013]. In NFM, the initial weights α(0) are uniformly initialized as (1/3, 1/3, 1/3) to ensure equal initial contributions of accuracy, fairness, and contrastive losses. The codes are available at https://github.com/getachAnGe/BAMFair.

5.2 Experimental Results

Performance Comparison (RQ1)

Table 2 presents results across four datasets with race and gender as sensitive attributes. We observe that: (1) BAMFair consistently outperforms all baselines in terms of fairness. This superiority stems from GBA's effective subgroup alignment (addressing CH1) and NFM's optimization of parameter update directions across conflicting losses (addressing CH2). (2) BAMFair achieves the highest accuracy among all fairness-optimizing baselines. Accuracy is slightly lower than Default because we sacrifice marginal performance for fairness. However, this drop remains minimal, and BAMFair still provides better accuracy than all other fairness-optimizing baselines. (3) In-processingmethods achieve a better trade-off between accuracy and fairness. In contrast, pre-processing methods often show unstable performance across datasets, as modifying raw data can lead to the loss of critical feature information. Methods like MAAT, FairGP, and BAMFair dynamically integrate fairness constraints into representation learning, making them more robust in complex scenarios.

To verify if BAMFair generates fairer distributions, we visualize model outputs on Compas in Figure 3. While the Default model exhibits clear bias (e.g., higher distance (S) for Black Males), BAMFair produces nearly identical distributions across subgroups by minimizing Sinkhorn distances, effectively achieving group fairness.

Ablation Study (RQ2)

To study the effects of individual components, we conduct ablation study on the Compas dataset, with results presented in Table 3. The improvement boost from incorporating GBA proves the effectiveness of aligning subgroups with the barycenter. Adding Lcon further improves results, validating that contrastive learning strengthens subgroup representation consistency. Finally, the performance improvement through augmenting NFM, (i.e., the complete BAMFair framework) achieves the best balance, proving its ability to reach global optimality by reconciling multi-objective conflicts.

Analysis of Scalability (RQ3)

Results with three sensitive attributes highlight the scalability of BAMFair. As shown in Figure 5, BAMFair maintains superior performance while scaling efficiently, making it highly suitable for complex real-world applications. As shown in Figure 4, BAMFair consistently achieves the lowest time cost compared to all scalable methods, regardless of whether two or three sensitive attributes are used. Meanwhile, our method demonstrates a much smaller increase in computational cost as the number of sensitive attributes grows. For methods like Fair-SMOTE and FairMask, the time spikes significantly by 141.42% and 156.66% respectively. In contrast, the time cost of BAMFair grows by only 46.88%, which proves the scalability of BAMFair.

Comparison with other MTL (RQ4)

To validate NFM, we compare several main multiple task learning (MTL) methods [Yu et al., 2024] on the Compas dataset. As shown in Table 4, NFM achieves the best performance. Among these methods, MGDA [Sener and Koltun, 2018] biases toward fairness due to its smaller loss scale. Other MTL methods [Yu et al., 2020; Liu et al., 2021], on the other hand, lean towards larger scaled accuracy loss, resulting in poor fairness performance. Although CAgrad gets higher accuracy, its fairness performance is notably worse than that of NFM. Only NFM manages to strike a good balance between fairness and accuracy.

6 Conclusion

To achieve fairness across multiple sensitive attributes, this paper proposes BAMFair. It utilizes GBA to align subgroups with a global barycenter, reducing alignment complexity while enhancing fairness and scalability, and NFM to resolve gradient conflicts. Across four publicly available realworld datasets, BAMFair outperforms state-of-the-art methods in terms of accuracy and fainess.

Acknowledgments

The work was supported by the National Natural Science Foundation of China (No.62572486), Natural Science Foundation of Shandong Province (No.ZR2023MF007), European Union under the Horizon Europe programme (grant numbers: 101135930, 101168438), in part by the UKRI through the UK Government's Horizon Europe funding Guarantee (grant numbers: 10119564, 10126241).