MagicTailor: Component-Controllable Personalization in Text-to-Image Diffusion Models

Zhou, Donghao; Huang, Jiancheng; Bai, Jinbin; Wang, Jiaze; Chen, Hao; Chen, Guangyong; Hu, Xiaowei; Heng, Pheng-Ann

doi:10.24963/ijcai.2025/1136

MagicTailor: Component-Controllable Personalization in Text-to-Image Diffusion Models

Donghao Zhou, Jiancheng Huang, Jinbin Bai, Jiaze Wang, Hao Chen, Guangyong Chen, Xiaowei Hu, Pheng-Ann Heng

Download BibTeX Cite

Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. AI, Arts & Creativity. Pages 10225-10233.

https://doi.org/10.24963/ijcai.2025/1136

Keywords

Theory and philosophy of arts and creativity in AI systems: Autonomous creative or artistic AI
Methods and resources: Machine learning, deep learning, neural models, reinforcement learning
Application domains: Images, movies and visual arts
Theory and philosophy of arts and creativity in AI systems: Computational paradigms, architectures and models for creativity
Methods and resources: AI systems for ideation

Abstract

Text-to-image diffusion models can generate high-quality images but lack fine-grained control of visual concepts, limiting their creativity. Thus, we introduce component-controllable personalization, a new task that enables users to customize and reconfigure individual components within concepts. This task faces two challenges: semantic pollution, where undesired elements disrupt the target concept, and semantic imbalance, which causes disproportionate learning of the target concept and component. To address these, we design MagicTailor, a framework that uses Dynamic Masked Degradation to adaptively perturb unwanted visual semantics and Dual-Stream Balancing for more balanced learning of desired visual semantics. The experimental results show that MagicTailor achieves superior performance in this task and enables more personalized and creative image generation.

Donghao Zhou, Jiancheng Huang, Jinbin Bai, Jiaze Wang, Hao Chen, Guangyong Chen, Xiaowei Hu, and Pheng-Ann Heng. 2025. MagicTailor: Component-Controllable Personalization in Text-to-Image Diffusion Models. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25. IJCAI, 10225–10233. https://doi.org/10.24963/ijcai.2025/1136

@inproceedings{ijcai2025p1136,
  title     = {MagicTailor: Component-Controllable Personalization in Text-to-Image Diffusion Models},
  author    = {Zhou, Donghao and Huang, Jiancheng and Bai, Jinbin and Wang, Jiaze and Chen, Hao and Chen, Guangyong and Hu, Xiaowei and Heng, Pheng-Ann},
  booktitle = {Proceedings of the Thirty-Fourth International Joint Conference on
               Artificial Intelligence, {IJCAI-25}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {James Kwok},
  pages     = {10225--10233},
  year      = {2025},
  month     = {8},
  note      = {AI, Arts & Creativity},
  doi       = {10.24963/ijcai.2025/1136},
  url       = {https://doi.org/10.24963/ijcai.2025/1136},
}

TY  - CPAPER
TI  - MagicTailor: Component-Controllable Personalization in Text-to-Image Diffusion Models
AU  - Zhou, Donghao
AU  - Huang, Jiancheng
AU  - Bai, Jinbin
AU  - Wang, Jiaze
AU  - Chen, Hao
AU  - Chen, Guangyong
AU  - Hu, Xiaowei
AU  - Heng, Pheng-Ann
T2  - Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2025
SP  - 10225
EP  - 10233
DO  - 10.24963/ijcai.2025/1136
UR  - https://doi.org/10.24963/ijcai.2025/1136
ER  -