{"type":"ScholarlyArticle","title":"Balancing Invariant and Specific Knowledge for Domain Generalization with Online Knowledge Distillation","authors":[{"name":"Di Zhao","slug":"di-zhao","given":"Di","family":"Zhao","affiliation":""},{"name":"Jingfeng Zhang","slug":"jingfeng-zhang","given":"Jingfeng","family":"Zhang","affiliation":""},{"name":"Hongsheng Hu","slug":"hongsheng-hu","given":"Hongsheng","family":"Hu","affiliation":""},{"name":"Philippe Fournier-Viger","slug":"philippe-fournier-viger","given":"Philippe","family":"Fournier-Viger","affiliation":""},{"name":"Gillian Dobbie","slug":"gillian-dobbie","given":"Gillian","family":"Dobbie","affiliation":""},{"name":"Yun Sing Koh","slug":"yun-sing-koh","given":"Yun Sing","family":"Koh","affiliation":""}],"abstract":"Recent research has demonstrated the effectiveness of knowledge distillation in Domain Generalization. However, existing approaches often overlook domain-specific knowledge and rely on an offline distillation strategy, limiting the effectiveness of knowledge transfer. To address these limitations, we propose Balanced Online knowLedge Distillation (BOLD). BOLD leverages a multi-domain expert teacher model, with each expert specializing in a specific source domain, enabling the student to distill both domain-invariant and domain-specific knowledge. We incorporate the Pareto optimization principle and uncertainty weighting to balance these two types of knowledge, ensuring simultaneous optimization without compromising either. Additionally, BOLD employs an online knowledge distillation strategy, allowing the teacher and student to learn concurrently. This dynamic interaction enables the teacher to adapt based on student feedback, facilitating more effective knowledge transfer. Extensive experiments on seven benchmarks demonstrate that BOLD outperforms state-of-the-art methods. Furthermore, we provide theoretical insights that highlight the importance of domain-specific knowledge and the advantages of uncertainty weighting.","doi":"10.24963/ijcai.2025/272","url":"https://ijcai.org/proceedings/2025/272","pdf":"https://ijcai.org/proceedings/2025/0272.pdf","markdown":"https://ijcai.org/proceedings/2025/272.md","bibtex":"https://ijcai.org/proceedings/2025/bibtex/272","volume":{"slug":"2025","title":"Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2025","url":"https://ijcai.org/proceedings/2025/","year":2025,"location":"Montreal, Canada","dates":"16-22 August 2025","editors":"James Kwok","isbn":"978-1-956792-06-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2025"},"section":"Computer Vision","track":"Main Track","number":272,"pages":{"first":2440,"last":2448},"keywords":["Computer Vision: CV: Transfer, low-shot, semi- and un- supervised learning","Machine Learning: ML: Foundation models"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}