QA-MDT: Quality-aware Masked Diffusion Transformer for Enhanced Music Generation

Li, Chang; Wang, Ruoyu; Liu, Lijuan; Du, Jun; Sun, Yixuan; Guo, Zilu; Zhang, Zhengrong; Jiang, Yuan; Gao, Jianqing; Ma, Feng

doi:10.24963/ijcai.2025/1126

QA-MDT: Quality-aware Masked Diffusion Transformer for Enhanced Music Generation

Chang Li, Ruoyu Wang, Lijuan Liu, Jun Du, Yixuan Sun, Zilu Guo, Zhengrong Zhang, Yuan Jiang, Jianqing Gao, Feng Ma

Download BibTeX Cite

Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. AI, Arts & Creativity. Pages 10135-10143.

https://doi.org/10.24963/ijcai.2025/1126

Keywords

Application domains: General
Application domains: Music and sound
Theory and philosophy of arts and creativity in AI systems: Autonomous creative or artistic AI
Methods and resources: Machine learning, deep learning, neural models, reinforcement learning

Abstract

Text-to-music (TTM) generation, which converts textual descriptions into audio, opens up innovative avenues for multimedia creation.

Achieving high quality and diversity in this process demands extensive, high-quality data, which are often scarce in available datasets. Most open-source datasets frequently suffer from issues like low-quality waveforms and low text-audio consistency, hindering the advancement of music generation models.

To address these challenges, we propose a novel quality-aware training paradigm for generating high-quality, high-musicality music from large-scale, quality-imbalanced datasets. Additionally, by leveraging unique properties in the latent space of musical signals, we adapt and implement a masked diffusion transformer (MDT) model for the TTM task, showcasing its capacity for quality control and enhanced musicality. Furthermore, we introduce a three-stage caption refinement approach to address low-quality captions' issue. Experiments show state-of-the-art (SOTA) performance on benchmark datasets including MusicCaps and the Song-Describer Dataset with both objective and subjective metrics.

Demo audio samples are available at https://qa-mdt.github.io/, code and pretrained checkpoints are open-sourced at https://github.com/ivcylc/OpenMusic.

Chang Li, Ruoyu Wang, Lijuan Liu, Jun Du, Yixuan Sun, Zilu Guo, Zhengrong Zhang, Yuan Jiang, Jianqing Gao, and Feng Ma. 2025. QA-MDT: Quality-aware Masked Diffusion Transformer for Enhanced Music Generation. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25. IJCAI, 10135–10143. https://doi.org/10.24963/ijcai.2025/1126

@inproceedings{ijcai2025p1126,
  title     = {QA-MDT: Quality-aware Masked Diffusion Transformer for Enhanced Music Generation},
  author    = {Li, Chang and Wang, Ruoyu and Liu, Lijuan and Du, Jun and Sun, Yixuan and Guo, Zilu and Zhang, Zhengrong and Jiang, Yuan and Gao, Jianqing and Ma, Feng},
  booktitle = {Proceedings of the Thirty-Fourth International Joint Conference on
               Artificial Intelligence, {IJCAI-25}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {James Kwok},
  pages     = {10135--10143},
  year      = {2025},
  month     = {8},
  note      = {AI, Arts & Creativity},
  doi       = {10.24963/ijcai.2025/1126},
  url       = {https://doi.org/10.24963/ijcai.2025/1126},
}

TY  - CPAPER
TI  - QA-MDT: Quality-aware Masked Diffusion Transformer for Enhanced Music Generation
AU  - Li, Chang
AU  - Wang, Ruoyu
AU  - Liu, Lijuan
AU  - Du, Jun
AU  - Sun, Yixuan
AU  - Guo, Zilu
AU  - Zhang, Zhengrong
AU  - Jiang, Yuan
AU  - Gao, Jianqing
AU  - Ma, Feng
T2  - Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2025
SP  - 10135
EP  - 10143
DO  - 10.24963/ijcai.2025/1126
UR  - https://doi.org/10.24963/ijcai.2025/1126
ER  -