Balancing Invariant and Specific Knowledge for Domain Generalization with Online Knowledge Distillation

Zhao, Di; Zhang, Jingfeng; Hu, Hongsheng; Fournier-Viger, Philippe; Dobbie, Gillian; Koh, Yun Sing

doi:10.24963/ijcai.2025/272

Balancing Invariant and Specific Knowledge for Domain Generalization with Online Knowledge Distillation

Di Zhao, Jingfeng Zhang, Hongsheng Hu, Philippe Fournier-Viger, Gillian Dobbie, Yun Sing Koh

Download BibTeX Cite

Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. Main Track. Pages 2440-2448.

https://doi.org/10.24963/ijcai.2025/272

Keywords

Computer Vision: CV: Transfer, low-shot, semi- and un- supervised learning
Machine Learning: ML: Foundation models

Abstract

Recent research has demonstrated the effectiveness of knowledge distillation in Domain Generalization. However, existing approaches often overlook domain-specific knowledge and rely on an offline distillation strategy, limiting the effectiveness of knowledge transfer. To address these limitations, we propose Balanced Online knowLedge Distillation (BOLD). BOLD leverages a multi-domain expert teacher model, with each expert specializing in a specific source domain, enabling the student to distill both domain-invariant and domain-specific knowledge. We incorporate the Pareto optimization principle and uncertainty weighting to balance these two types of knowledge, ensuring simultaneous optimization without compromising either. Additionally, BOLD employs an online knowledge distillation strategy, allowing the teacher and student to learn concurrently. This dynamic interaction enables the teacher to adapt based on student feedback, facilitating more effective knowledge transfer. Extensive experiments on seven benchmarks demonstrate that BOLD outperforms state-of-the-art methods. Furthermore, we provide theoretical insights that highlight the importance of domain-specific knowledge and the advantages of uncertainty weighting.

Di Zhao, Jingfeng Zhang, Hongsheng Hu, Philippe Fournier-Viger, Gillian Dobbie, and Yun Sing Koh. 2025. Balancing Invariant and Specific Knowledge for Domain Generalization with Online Knowledge Distillation. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25. IJCAI, 2440–2448. https://doi.org/10.24963/ijcai.2025/272

@inproceedings{ijcai2025p272,
  title     = {Balancing Invariant and Specific Knowledge for Domain Generalization with Online Knowledge Distillation},
  author    = {Zhao, Di and Zhang, Jingfeng and Hu, Hongsheng and Fournier-Viger, Philippe and Dobbie, Gillian and Koh, Yun Sing},
  booktitle = {Proceedings of the Thirty-Fourth International Joint Conference on
               Artificial Intelligence, {IJCAI-25}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {James Kwok},
  pages     = {2440--2448},
  year      = {2025},
  month     = {8},
  note      = {Main Track},
  doi       = {10.24963/ijcai.2025/272},
  url       = {https://doi.org/10.24963/ijcai.2025/272},
}

TY  - CPAPER
TI  - Balancing Invariant and Specific Knowledge for Domain Generalization with Online Knowledge Distillation
AU  - Zhao, Di
AU  - Zhang, Jingfeng
AU  - Hu, Hongsheng
AU  - Fournier-Viger, Philippe
AU  - Dobbie, Gillian
AU  - Koh, Yun Sing
T2  - Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2025
SP  - 2440
EP  - 2448
DO  - 10.24963/ijcai.2025/272
UR  - https://doi.org/10.24963/ijcai.2025/272
ER  -