FedAMB: Adaptive Modality Balancing for Dominance-Robust Multimodal Federated Distillation

1 Introduction

The expanding application of artificial intelligence has increased the demand for robust data protection mechanisms. Federated Learning (FL) has emerged as a pivotal solution to this challenge, particularly in privacy-sensitive domains such as healthcare [Xu et al., 2021], finance [Long et al., 2020], and personalized services [Mammen, 2021]. In a typical FL setting, a global model is distributed from a central server to multiple clients [McMahan et al., 2017], where it is trained locally using the private data of the clients. Instead of sharing sensitive raw data, clients transmit model updates, such as weights [Kairouz et al., 2021] or gradients [Wen et al., 2017], back to the server. The server aggregates these updates to refine the global model and initiates subsequent training rounds. This collaborative paradigm enables multiple organizations to build robust models from diverse data sources without exposing confidential information [Rieke et al., 2020]. Despite their success, conventional FL approaches that rely on modelupdate exchanges suffer from several limitations. Transmitting the full model parameters of large-scale networks incurs a substantial communication overhead. Moreover, enforcing a homogeneous model architecture across all clients restricts flexibility in the presence of heterogeneous data distributions and computational resources [Mora et al., 2022]. The direct sharing of model updates also raises concerns regarding information leakage and intellectual property protection [Yin et al., 2021]. To alleviate these issues, Federated Knowledge Distillation (Fed-KD) has been proposed as a promising alternative [Li et al., 2024]. Fed-KD exchanges condensed knowledge, such as intermediate features or output logits, rather than full model parameters, thereby reducing the communication costs and mitigating privacy risks.

∗Corresponding authors

Meanwhile, in real-world FL applications, data are often inherently multimodal and comprise multiple information sources, such as images, text, and speech. Learning from multiple modalities enables models to capture complementary information and achieve superior performance compared with unimodal approaches. Motivated by this, Multimodal Federated Learning (MFL) has emerged as a promising research direction that combines multimodal representation learning with the privacy-preserving advantages of FL [Che et al., 2023]. However, the multimodal-federated optimization interaction introduces challenges that do not occur in unimodal FL. A primary issue is modality dominance, where a single strong or easier-to-learn modality disproportionately governs the learning process, suppressing complementary signals from other modalities and leading to suboptimal multimodal fusion [Wang et al., 2020] (Fig. 1, right). In practical terms, a client model may appear to perform well by relying mostly on its locally dominant modality, while the remaining modalities are under-trained and become unreliable under missing-modality inference. This problem is significantly exacerbated in federated environments with nonindependent and identically distributed (Non-IID) data [Zhao et al., 2018], where different clients may develop different modality-specific biases. Consequently, in Fed-KD-based MFL, these client-specific biases can be propagated and amplified through server-side aggregation, contaminating the global teacher and ultimately degrading the performance and robustness of the global model [Ouyang et al., 2023].

Another critical challenge arises from missing modalities during deployment. Owing to sensor failures or resource constraints, one or more modalities may be unavailable at inference time [Wu et al., 2024]. As a result, clients exhibit inconsistent modality configurations (Figure 1, left). Therefore, a practical MFL framework must deliver reliable performance even under unimodal inference without incurring communication overheads [Wang et al., 2024]. Prior studies have attempted to mitigate these challenges; however, existing methods often lack an aggregation-aware perspective. In particular, they overlook how dominance-induced client bias can distort server aggregation and contaminate the global teacher, thereby propagating biased knowledge back to clients. Moreover, prior dominance mitigation typically assumes a static imbalance and applies uniform interventions (e.g., fixed weighting or modality dropping), whereas dominance in MFL is inherently state-dependent across clients and rounds. As a result, indiscriminate regulation may unnecessarily disrupt stable multimodal fusion.

To address these challenges, we propose a novel MFL framework built upon the Fed-KD paradigm, called Federated Adaptive Modality Balancing (FedAMB). At the client level, Selective-Modality Regulation (SMR) mitigates dominance at its source by selectively regulating biased local learning dynamics. At the global stage, Component-wise Modality Distillation (CMD) decomposes and regulates the flow of global knowledge, preventing client-level fusion bias from being propagated through the global teacher and directly strengthening unimodal representations. Through the joint application of SMR and CMD, FedAMB enhances the expressiveness of unimodal representations while stabilizing multimodal fusion under heterogeneous and incomplete modality settings. Extensive experiments on the CREMAD, AVE, and UR-FUNNY datasets, conducted under varying degrees of data heterogeneity, demonstrate that FedAMB consistently outperforms existing methods in terms of multimodal accuracy and unimodal robustness. Our contributions are summarized as follows:

• First contamination-aware view of modality dominance in Fed-KD MFL: To the best of our knowledge, this is the first work to cast modality dominance as global-teacher contamination in Fed-KD-based MFL, showing that dominance-induced client bias is amplified through server aggregation and degrades both multimodal accuracy and unimodal robustness.
• FedAMB: stage-wise dominance mitigation with selective intervention and branch-wise distillation: We propose FedAMB, which (i) activates SMR only when dominance destabilizes local learning and (ii) introduces CMD to distill global knowledge into each modality branch, preventing fusion-biased teacher signals from suppressing unimodal representations.
• Robust improvements under realistic FL conditions: Across CREMA-D, AVE, and UR-FUNNY, FedAMB consistently improves multimodal performance and unimodal robustness under heterogeneous Non-IID partitions and missing-modality settings, outperforming strong baselines by a stable margin.

2 RelatedWork

2.1 Federated Learning Paradigms

FL is a privacy-enhancing technique that enables collaborative model training on decentralized data [Li et al., 2020a]. The most common approach involves clients sharing parameter updates, such as weights [Karimireddy et al., 2020] or gradients [Yuan and Ma, 2020] from their locally trained models with a server to build a single global model. These conventional FL approaches, which are based on exchanging global models, suffer from several inherent weaknesses [Li et al., 2020b]. First, the increasing size of deep neural networks leads to prohibitive communication overheads when model weights are directly exchanged [Lin et al., 2018; Alistarh et al., 2017]. Second, it imposes a rigid requirement for a homogeneous model architecture across all participating clients [Diao et al., 2021]. Finally, the direct exchange of model updates creates significant security vulnerabilities, exposing the system to threats such as model inversion attacks, which can potentially reconstruct private training data [Fraboni et al., 2021]. Communication, architectural flexibility, and security challenges are key factors that impede the practical deployment of FL.

2.2 Knowledge Distillation with Proxy Dataset

To overcome the limitations of update-sharing FL, Fed-KD was proposed as an effective alternative [Qin et al., 2024]. The core idea of Fed-KD is to exchange "knowledge," such as model outputs (logits) or features [Yang et al., 2023] from intermediate layers, instead of heavy model parameters [Lin et al., 2020]. A shared public dataset is necessary for effectively exchanging knowledge among clients who cannot share their private data with others. Therefore, a concept of a proxy dataset was proposed that all participants can access [Hinton et al., 2015a]. This dataset enables clients to generate predictions on identical inputs, which can then be aggregated by the server to form global teacher knowledge for distillation [Hinton et al., 2015b].

2.3 Multimodal Federated Learning

The recent explosion of data from the Internet, sensors, and mobile devices has significantly increased the importance of multimodal learning, which is a field dedicated to leveraging diverse data types in unison [Baltruˇsaitis et al., 2018]. MFL has arisen from this background as a novel research field that integrates the privacy advantages of FL with the powerful analytical capabilities of multimodal learning [Yu et al., 2023]. However, MFL faces a set of unique and multifaceted problems that are distinct from those in unimodal FL [Adam et al., 2025]. One of these is modality heterogeneity, in which clients possess different sets of modalities, leading to architectural discrepancies that hinder the aggregation of models. Furthermore, MFL often suffers from modality dominance, a phenomenon in which a single dominant modality can overwhelm the learning process and suppress valuable signals from other modalities. Several studies have attempted to address these challenges in MFL. One line of research focuses on handling incomplete or missing modalities. For instance, some approaches utilize contrastive learning to learn robust multimodal representations even when certain modalities are unavailable [Che et al., 2024], while others investigate methods for generating or restoring missing-modality data at the client level for training [Xiong et al., 2023]. Another line of research addresses the modality heterogeneity and imbalance. One study proposed the use of submodular functions to promote modality diversity during client selection and introduced a modal enhancement loss to facilitate the learning of weak modalities [Fan et al., 2024]. However, this approach is based on conventional FL, which exchanges model updates.Existing Fed-KD methods can falter in MFL because dominance-biased client predictions may be aggregated into a fusion-biased global teacher, which then suppresses weak-modality learning again during distillation [Sarfraz et al., 2021]. Different from parameter-aggregation-based MFL methods, FedAMB follows a Fed-KD protocol that aggregates proxy predictions as the primary communication signal, while exchanging lightweight class-wise prototypes for dominance identification. With an aggregation-aware stagewise design (SMR+CMD), FedAMB mitigates this specific teacher-contamination failure mode and improves both multimodal performance and unimodal robustness.

3 Methodology

3.1 ProblemFormulation

We consider a MFL system with K clients and a modality set M. Client k holds a private labeled dataset

where Nk is the number of samples, and yk i ∈ {1, . . . ,Nc} is the class label. We split Dk into a training set Dtrain k and a test set Dtest k . Following the federated knowledge distillation (Fed-KD) paradigm, we assume an unlabeled proxy dataset that is accessible to all clients:

The goal is to collaboratively improve each client model Cθk without sharing raw private data, such that the resulting model achieves (i) high multimodal accuracy under the available modalities and (ii) robust unimodal performance when some modalities are absent at inference.

Algorithm 1 FedAMB (Per Round)

Input: total rounds T, dominance threshold γ, fusioninstability threshold τ , KD temperature Tkd

Output: client models {θk}

• 1: Server maintains global teacher Z G and global prototypes {P G}
• 2: for t = 1 to T do
• 3: Server samples clients St and broadcasts (Z G, {P G})
• 4: for each client k ∈ St do
• 5: Compute PAcck,t(m) using a private sample and {P G}
• 6: M k,t D ←argmaxmPAcck,t(m), M k,t W ←arg minm PAcck,t(m)
• 7: sk,t ← [ PAcck,t(M k,t D ) > γ · PAcck,t(M k,t W ) ]
• 8: Run local training with SMR if sk,t = 1; else standard multimodal training
• 9: Upload proxy logits zk proxy and prototype stats to server
• 10: end for
• 11: Server aggregates {zk proxy } to update Z G and aggregates prototypes to update {P G}
• 12: Server broadcasts updated Z G
• 13: for each client k ∈ St do
• 14: Compute proxy logits (zkM , zkW )
• 15: LkCMD = ∑u∈{M,W} KL(zku ,Z G)
• 16: Update θk by minimizing LkCMD
• 17: end for
• 18: end for
• 19: return {θk}

3.2 FedAMB: Overall Framework

FedAMB is a stage-wise Fed-KD framework that mitigates dominance-induced client bias before aggregation and controls the propagation of global knowledge back to each modality branch. Communication round t consists of the following steps: Figure 2 provides an overview of the pipeline, and Algorithm 1 summarizes the detailed round procedure. Specifically, (0) Prototype Upload and Aggregation: each selected client computes modality- and class-specific local prototypes, uploads (P k,c m,t ,N k,c ) to the server, and the server aggregates them into global prototypes {P G,c m,t } and broadcasts them back for prototype-based dominance identification; (1) Selective-Modality Regulation (SMR): using {P G,c m,t }, client k estimates the dominance state sk,t and, if dominated, suppresses the dominant modality by training with a weakmodality logit (masking the dominant modality); (2) Proxy Logit Upload: clients compute multimodal logits on the proxy samples and upload them as distilled knowledge; (3) Teacher Construction and Distribution: the server aggregates uploaded logits to construct the global teacher zG t and broadcasts it to clients; and (4) Component-wise Modality Distillation (CMD): each client distills the global teacher into both its multimodal logits and weak unimodal logits, strengthening the weak branch while preventing fusion-induced bias from being propagated through global aggregation.

3.3 Dominance Identification as a State Variable

Let m ∈ M denote modality. Client k maintains a modalityspecific encoder E km that extracts features f km,i,t = E km(xkm,i ) at round t. For each class c, client k computes the local prototype as

where N k,c is the number of samples of class c at client k (computed on Dtrain k ). Clients upload {(P k,c m,t ,N k,c )} to the server, which forms a class-weighted global prototype:

Prototype-based accuracy. Given global prototypes {P G,c m,t }c, client k assigns a prototype label using cosine similarity as follows:

To avoid ambiguity and keep the computation lightweight, we estimate the dominance metric on a sampled labeled minibatch Bk,t ⊂ Dtrain k :

Dominance state. We define the dominant and weak modalities for client k at round t as

and the dominance indicator

where γ > 1 is the threshold.

3.4 Selective-Modality Regulation (SMR)

Let F k be the fusion head producing multimodal logits from modality features, and let Hkm be an auxiliary unimodal head for modality m. At round t, for a sample i, the multimodal logit is

When sk,t = 0 (non-dominant state), the client performs standard multimodal training with zkM,i,t .

When sk,t = 1 (dominant state), SMR suppresses the dominant modality by masking its features and constructing an alternative training logit for the weak modality. We define masked features

and compute a fusion logit without the dominant modality as follows:

We also compute the weak-modality unimodal logit:

Let Acck,t M and Acck,t MD denote the training accuracy of the multimodal prediction and the dominant-modality prediction, respectively. To avoid over-intervention within the dominant states, we activate a fusion-instability gate only when multimodal fusion becomes less reliable than the dominantmodality prediction:

Then, we incorporate the masked-fusion signal ˜ zkM,i,t as a corrective term and construct the weak-modality training logit in a single form:

We choose the local training logit as

and minimize the local cross-entropy loss

3.5 Component-wise Modality Distillation (CMD)

After the SMR-based local updates, the clients participate in Fed-KD by computing the proxy logits. Client k uploads its multimodal proxy logits {zkM,t (x)}x∈Dproxy . The server constructs a global teacher by averaging the logits as follows:

where St is the set of participating clients in round t.

The global teacher serves as a soft ensemble target capturing class structures and unbiased boundaries, rather than requiring exact reproduction of multimodal data. Thus, distillation regularizes unimodal representations toward this structure instead of transferring full multimodal information. This allows CMD to stabilize weak-modality learning without assuming teacher-student capacity alignment. Each client then performs distillation at a fixed temperature Tkd > 0:

Additionally, the same teacher is distilled into the weakmodality branch:

The proxy loss is

The CMD explicitly regulates the transfer of global knowledge to each modality component, strengthening unimodal representations while avoiding the blind propagation of fusion bias through the global teacher.

4 Experiments

Datasets We evaluate FedAMB on two audio-visual datasets and one audio-visual-text dataset. For audio-visual learning, we use CREMA-D [Cao et al., 2014] (emotion recognition) and AVE [Tian et al., 2018] (event detection), where modality dominance is observed, thus providing a suitable testbed for our dominance-aware MFL setting. To further assess scalability beyond two modalities, we additionally use UR-FUNNY, an audio-visual-text benchmark, to evaluate FedAMB under N-modal learning.

Implementation Details We model client-level statistical heterogeneity using Dirichlet partitioning over class labels and define the non-IID setting with α = 1.0. Unless otherwise specified, we run federated learning with 10 clients for 100 communication rounds, where 4 clients are randomly selected per round, and set the dominance threshold to γ = 1.5. To simulate an in-domain FL environment under the Fed-KD protocol, we sample 20% of each dataset as a public, unlabeled proxy set for knowledge exchange, and distribute the remaining 80% across clients as private local datasets.

Baselines To evaluate, we include FedMD [Li and Wang, 2019], FedProto [Tan et al., 2022], FedIoT-KD [Zhang et al., 2021], and DaFKD [Wang et al., 2023] as representative FedKD and prototype-based baselines for MFL. In addition, we compare with Random Modal Drop (RMD) [Alfasly et al., 2022] and FedPMR [Fan et al., 2023] as modality-dominance mitigation baselines.

4.1 Evaluation Results

Table 1 reports the performance comparison of CREMAD, AVE, and UR-FUNNY under IID and non-IID settings. Overall, FedAMB consistently outperforms strong baselines across all datasets, demonstrating robust gains under both homogeneous and heterogeneous client distributions. In particular, FedAMB improves over the best baseline by +5.29/+1.64 points on CREMA-D (IID/Non-IID), +0.30/+0.15 points on AVE, and +0.13/+0.56 points on URFUNNY, respectively. These results indicate that FedAMB generalizes beyond audio-visual settings to N-modality scenarios and yields stable improvements under federated heterogeneity. Unlike RMD and FedPMR, FedAMB explicitly suppresses dominance-induced bias in aggregated logit knowledge via SMR and CMD, leading to a more stable and robust performance under heterogeneity. To further analyze the training dynamics, we visualize the convergence behavior of all methods for each dataset. Figure 4 presents the test accuracy curves for the communication rounds. Notably, although FedAMB employs a state-dependent loss, it remains stable throughout the training and consistently converges under federated heterogeneity.

4.2 Unimodal Performance Analysis

To further investigate the effectiveness of FedAMB, we conduct a unimodal performance analysis on the CREMA-D dataset. This analysis evaluates the model's accuracy on individual audio and visual modalities by replacing the feature vector from the other modality with zero vectors. Table 2 presents the accuracy of unimodal inferences. In the IID setting, FedAMB improves the visual modality by more than +11.47 points and achieves the highest audio accuracy in both IID (40.78%) and non-IID (39.52%) settings. The consistent top performance across both modalities, especially the improvement in the weak modality (visual), underscores the capability of FedAMB to effectively leverage and balance information from different sources. This holds for both IID and the more challenging non-IID settings, further validating the robustness and efficacy of the FedAMB framework.

4.3 Robustness under a Public Proxy Dataset

To assess the practicality of FedAMB when an in-domain proxy is unavailable, we consider a domain-mismatched public proxy scenario where all methods distill knowledge from CMU-MOSEI [Zadeh et al., 2018] (Table 3). Despite the distribution shift between the proxy and private client data, FedAMB remains consistently competitive and achieves the best performance across both datasets and client distributions. Specifically, FedAMB improves over the strongest baseline by +4.74/+0.71 points on CREMA-D (IID/Non-IID) and by +0.93/+0.97 points on AVE. This suggests that FedAMB is robust to proxy mismatches and can still suppress dominanceinduced bias even when the global teacher is constructed from an external public dataset, supporting its deployment in realistic Fed-KD settings.

4.4 Ablation Study

Table 4 presents an ablation study on CREMA-D and AVE by selectively enabling the SMR and CMD. As a baseline, we use the model trained without both SMR and CMD, which achieves 41.08/15.03 (Multi/Visual) on CREMA-D and 36.11/10.33 on AVE. Overall, SMR mainly improves weak-modality learning by mitigating modality dominance during local training, while its standalone effect on multimodal accuracy can vary across datasets. CMD is most effective when combined with SMR by distilling the aggregated teacher into modality-specific branches, thereby consolidating and amplifying the benefits of local regulation. Quantitatively, on CREMA-D, the full model improves multimodal accuracy by +9.03 points and visual unimodal accuracy by +19.09 points over the baseline, and on AVE, it yields a +0.63 point gain in multimodal accuracy. These results support the complementary roles of the SMR in stabilizing local multimodal optimization and CMD in modality-aware global knowledge transfer. We further examine the dual-head design in SMR (Table 5). The results show that introducing the dual-head consistently improves multimodal accuracy across datasets, indicating that the additional modality-specific supervision helps stabilize multimodal fusion and enhances the effectiveness of dominance-aware regulation.

5 Discussion

Although FedAMB improves performance and stability under heterogeneity, it currently relies on a proxy dataset for distillation, which may be infeasible to obtain in practice and may introduce domain mismatches. Simultaneously, sharing class-wise modality statistics (e.g., prototypes and counts) may reveal coarse information about client-level class distributions under an honest-but-curious server assumption. To mitigate this risk, FedAMB can be combined with standard privacy-enhancing mechanisms, such as secure aggregation, count binning/clipping, and DP-style perturbation of shared prototypes without changing the overall training protocol. Developing proxy-free distillation alternatives and systematically studying privacy utility trade-offs in prototype/logit exchanges remain important directions for future work.

6 Conclusion

To address modality dominance and unimodal fragility in Fed-KD-based MFL, we propose Federated Adaptive Modality Balancing (FedAMB). FedAMB is built on two complementary components: Selective-Modality Regulation (SMR), which mitigates dominance at its source by selectively regulating biased local dynamics in a state-aware manner, and Component-wise Modality Distillation (CMD), which decomposes and regulates global knowledge transfer to prevent the propagation of client-level fusion bias while directly strengthening unimodal representations. Overall, we formulate modality dominance as aggregation-induced contamination of global teacher knowledge and show that the joint application of SMR and CMD stabilizes multimodal fusion under heterogeneous and incomplete modality settings. Extensive experiments on CREMA-D, AVE, and UR-FUNNY demonstrate consistent gains in both multimodal accuracy and unimodal robustness over strong baselines.

Acknowledgements

This research was supported by the MSIT(Ministry of Science and ICT), Korea, under the ICAN (ICT Challenge and Advanced Network of HRD) support program (RS2024-00436934), the ITRC (Information Technology Research Center) grant (RS-2024-00436936), and the Graduate School of Virtual Convergence support program (RS-202300254129) supervised by the IITP (Institute of Information & communications Technology Planning & Evaluation).