{"type":"ScholarlyArticle","title":"ExpertDiff: Head-less Model Reprogramming with Diffusion Classifiers for Out-of-Distribution Generalization","authors":[{"name":"Jee Seok Yoon","slug":"jee-seok-yoon","given":"Jee Seok","family":"Yoon","affiliation":""},{"name":"Junghyo Sohn","slug":"junghyo-sohn","given":"Junghyo","family":"Sohn","affiliation":""},{"name":"Wootaek Jeong","slug":"wootaek-jeong","given":"Wootaek","family":"Jeong","affiliation":""},{"name":"Heung-Il Suk","slug":"heung-il-suk","given":"Heung-Il","family":"Suk","affiliation":""}],"abstract":"Vision-language models have achieved remarkable performance across various tasks by leveraging large-scale multimodal training data. However, their ability to generalize to out-of-distribution (OOD) domains requiring expert-level knowledge remains an open challenge. To address this, we investigate cross-domain transfer learning approaches for efficiently adapting diffusion classifiers to new target domains demanding expert-level domain knowledge. Specifically, we propose ExpertDiff, a head-less model reprogramming technique that optimizes the instruction-following abilities of text-to-image diffusion models via learnable prompts, while leveraging the diffusion classifier objective as a modular plug-and-play adaptor. Our approach eliminates the need for conventional output mapping layers (e.g., linear probes), enabling seamless integration with off-the-shelf diffusion frameworks like Stable Diffusion. We demonstrate the effectiveness of ExpertDiff on the various OOD datasets (i.e., medical and satellite imagery). Furthermore, we qualitatively showcase ExpertDiff’s ability to faithfully reconstruct input images, highlighting its potential for both downstream discriminative and upstream generative tasks. Our work paves the way for effectively repurposing powerful foundation models for novel OOD applications requiring domain expertise.","doi":"10.24963/ijcai.2025/764","url":"https://ijcai.org/proceedings/2025/764","pdf":"https://ijcai.org/proceedings/2025/0764.pdf","markdown":"https://ijcai.org/proceedings/2025/764.md","bibtex":"https://ijcai.org/proceedings/2025/bibtex/764","volume":{"slug":"2025","title":"Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2025","url":"https://ijcai.org/proceedings/2025/","year":2025,"location":"Montreal, Canada","dates":"16-22 August 2025","editors":"James Kwok","isbn":"978-1-956792-06-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2025"},"section":"Machine Learning","track":"Main Track","number":764,"pages":{"first":6866,"last":6874},"keywords":["Machine Learning: ML: Learnware/model reuse/transfer learning","Computer Vision: CV: Biomedical image analysis","Machine Learning: ML: Generative models","Computer Vision: CV: Transfer, low-shot, semi- and un- supervised learning"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}