{"type":"ScholarlyArticle","title":"Arrange, Inpaint, and Refine: Steerable Long-term Music Audio Generation and Editing via Content-based Controls","authors":[{"name":"Liwei Lin","slug":"liwei-lin","given":"Liwei","family":"Lin","affiliation":"Music X Lab , New York University Shanghai; Music X Lab , Mohamed bin Zayed University of Artificial Intelligence 3 C4DM , Queen Mary University of London"},{"name":"Gus Xia","slug":"gus-xia","given":"Gus","family":"Xia","affiliation":"Music X Lab , Mohamed bin Zayed University of Artificial Intelligence 3 C4DM , Queen Mary University of London; Music X Lab , New York University Shanghai"},{"name":"Yixiao Zhang","slug":"yixiao-zhang","given":"Yixiao","family":"Zhang","affiliation":""},{"name":"Junyan Jiang","slug":"junyan-jiang","given":"Junyan","family":"Jiang","affiliation":"Music X Lab , New York University Shanghai; Music X Lab , Mohamed bin Zayed University of Artificial Intelligence 3 C4DM , Queen Mary University of London"}],"abstract":"Controllable music generation plays a vital role in human-AI music co-creation. While Large Language Models (LLMs) have shown promise in generating high-quality music, their focus on autoregressive generation limits their utility in music editing tasks. To bridge this gap, To address this gap, we propose a novel approach leveraging a parameter-efficient heterogeneous adapter combined with a masking training scheme. This approach enables autoregressive language models to seamlessly address music inpainting tasks. Additionally, our method integrates frame-level content-based controls, facilitating track-conditioned music refinement and score-conditioned music arrangement. We apply this method to fine-tune MusicGen, a leading autoregressive music generation model. Our experiments demonstrate promising results across multiple music editing tasks, offering more flexible controls for future AI-driven music editing tools. The source codes and a demo page showcasing our work are available at https://kikyo-16.github.io/AIR.","doi":"10.24963/ijcai.2024/851","url":"https://ijcai.org/proceedings/2024/851","pdf":"https://ijcai.org/proceedings/2024/0851.pdf","markdown":"https://ijcai.org/proceedings/2024/851.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/851","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"AI, Arts & Creativity","track":"AI, Arts & Creativity","number":851,"pages":{"first":7690,"last":7698},"keywords":["Methods and resources: Machine learning, deep learning, neural models, reinforcement learning","Application domains: Music and sound"],"videos":[],"references":[{"position":1,"raw":"[Agostinelli et al., 2023] Andrea Agostinelli, Timo I Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Cail- lon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. MusicLM: Generating music from text. arXiv preprint arXiv:2301.11325, 2023.","title":"MusicLM: Generating music from text","authors":"Andrea Agostinelli, I Timo, Zalán Denk, Jesse Borsos, Mauro Engel, Antoine Verzetti, Qingqing Cail-Lon, Aren Huang, Adam Jansen, Marco Roberts, Tagliasacchi","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Böck et al., 2016] Sebastian Böck, Filip Korzeniowski, Jan Schlüter, Florian Krebs, and Gerhard Widmer. Madmom: A new python audio and music signal processing library. In Proceedings of the 24th ACM international conference on Multimedia, pages 1174-1178, 2016.","title":"Madmom: A new python audio and music signal processing library","authors":"Sebastian Böck, Filip Korzeniowski, Jan Schlüter, Florian Krebs, Gerhard Widmer","year":"2016","venue":"Proceedings of the 24th ACM international conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chang et al., 2021] Chin-Jui Chang, Chun-Yi Lee, and Yi- Hsuan Yang. Variable-length music score infilling via xl- net and musically specialized positional encoding. In Pro- ceedings of the 22nd Conference of the International Soci- ety for Music Information Retrieval, pages 97-104, 2021.","title":"Variable-length music score infilling via xlnet and musically specialized positional encoding","authors":"Chin-Jui Chang, Chun-Yi Lee, Yi-Hsuan Yang","year":"2021","venue":"Proceedings of the 22nd Conference of the International Society for Music Information Retrieval","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2020] Ke Chen, Cheng-i Wang, Taylor Berg- Kirkpatrick, and Shlomo Dubnov. Music sketchnet: Con- trollable music generation via factorized representations of pitch and rhythm. In Proceedings of the 21st Conference of the International Society for Music Information Retrieval, pages 77-84, 2020.","title":"Music sketchnet: Controllable music generation via factorized representations of pitch and rhythm","authors":"Ke Chen, Cheng-I Wang, Taylor Berg-Kirkpatrick, Shlomo Dubnov","year":"2020","venue":"Proceedings of the 21st Conference of the International Society for Music Information Retrieval","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Copet et al., 2023] Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music gen- eration. arXiv preprint arXiv:2306.05284, 2023.","title":"Simple and controllable music generation","authors":"Copet","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Défossez et al., 2022] Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural au- dio compression. arXiv preprint arXiv:2210.13438, 2022.","title":"High fidelity neural audio compression","authors":"Alexandre Défossez, Jade Copet, Gabriel Synnaeve, Yossi Adi","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Défossez et al., 2023] Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. Transactions on Machine Learning Research, 2023. Featured Certification, Reproducibility Certification.","title":"High fidelity neural audio compression","authors":"Alexandre Défossez, Jade Copet, Gabriel Synnaeve, Yossi Adi","year":"2023","venue":"Transactions on Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dhariwal et al., 2020] Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint arXiv:2005.00341, 2020.","title":"Jukebox: A generative model for music","authors":"Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, Ilya Sutskever","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gao et al., 2023] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304.15010, 2023.","title":"Llama-adapter v2: Parameter-efficient visual instruction model","authors":"Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Garcia et al., 2023] Hugo F Flores Garcia, Prem Seethara- man, Rithesh Kumar, and Bryan Pardo. Vampnet: Music generation via masked acoustic token modeling. In Ismir 2023 Hybrid Conference, 2023.","title":"Vampnet: Music generation via masked acoustic token modeling","authors":"Hugo F Flores Garcia, Prem Seetharaman, Rithesh Kumar, Bryan Pardo","year":"2023","venue":"Ismir 2023 Hybrid Conference","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Goto et al., 2002] Masataka Goto, Hiroki Hashiguchi, Takuichi Nishimura, and Ryuichi Oka. Rwc music database: Popular, classical and jazz music databases. In Proceedings of the 3rd Conference of the International Society for Music Information Retrieval, volume 2, pages 287-288, 2002.","title":"Rwc music database: Popular, classical and jazz music databases","authors":"Masataka Goto, Hiroki Hashiguchi, Takuichi Nishimura, Ryuichi Oka","year":"2002","venue":"Proceedings of the 3rd Conference of the International Society for Music Information Retrieval","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hu et al., 2021] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.","title":"Lora: Low-rank adaptation of large language models","authors":"J Edward, Yelong Hu, Phillip Shen, Zeyuan Wallis, Yuanzhi Allen-Zhu, Shean Li, Lu Wang, Weizhu Wang, Chen","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Huang and Yang, 2020] Yu-Siang Huang and Yi-Hsuan Yang. Pop music transformer: Beat-based modeling and generation of expressive pop piano compositions. In Pro- ceedings of the 28th ACM international conference on multimedia, pages 1180-1188, 2020.","title":"Pop music transformer: Beat-based modeling and generation of expressive pop piano compositions","authors":"Yang ; ] Yu-Siang Huang, Yi-Hsuan Huang, Yang","year":"2020","venue":"Proceedings of the 28th ACM international conference on multimedia","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Huang et al., 2018] Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer. arXiv preprint arXiv:1809.04281, 2018.","title":"Music transformer","authors":"Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, Douglas Eck","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ippolito et al., 2018] Daphne Ippolito, Anna Huang, Cur- tis Hawthorne, and Douglas Eck. Infilling piano perfor- mances. In NIPS Workshop on Machine Learning for Cre- ativity and Design, volume 2, page 5, 2018.","title":"Infilling piano performances","authors":"Daphne Ippolito, Anna Huang, Curtis Hawthorne, Douglas Eck","year":"2018","venue":"NIPS Workshop on Machine Learning for Creativity and Design","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Jiang et al., 2019] Junyan Jiang, Ke Chen, Wei Li, and Gus Xia. Large-vocabulary chord transcription via chord struc- ture decomposition. In Proceedings of the 20th Confer- ence of the International Society for Music Information Retrieval, pages 644-651, 2019.","title":"Large-vocabulary chord transcription via chord structure decomposition","authors":"Junyan Jiang, Ke Chen, Wei Li, Gus Xia","year":"2019","venue":"Proceedings of the 20th Conference of the International Society for Music Information Retrieval","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li and Liang, 2021] Xiang Lisa Li and Percy Liang. Prefix- tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021.","title":"Prefixtuning: Optimizing continuous prompts for generation","authors":"Lisa Xiang, Percy Li, Liang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lin et al., 2024] Liwei Lin, Gus Xia, Junyan Jiang, and Yix- iao Zhang. Content-based controls for music large lan- guage modeling, 2024.","title":"Content-based controls for music large language modeling","authors":"Liwei Lin, Gus Xia, Junyan Jiang, Yixiao Zhang","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2022] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Vol- ume 2: Short Papers), pages 61-68, 2022.","title":"P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks","authors":"Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, Jie Tang","year":"2022","venue":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Manilow et al., 2019] Ethan Manilow, Gordon Wichern, Prem Seetharaman, and Jonathan Le Roux. Cutting mu- sic source separation some slakh: A dataset to study the impact of training data quality and quantity. In 2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pages 45-49. IEEE, 2019.","title":"Cutting music source separation some slakh: A dataset to study the impact of training data quality and quantity","authors":"Ethan Manilow, Gordon Wichern, Prem Seetharaman, Jonathan Le Roux","year":"2019","venue":"IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Marafioti et al., 2019] Andrés Marafioti, Nathanaël Per- raudin, Nicki Holighaus, and Piotr Majdak. A context en- coder for audio inpainting. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(12):2362- 2372, 2019.","title":"A context encoder for audio inpainting","authors":"Marafioti","year":"2019","venue":"IEEE/ACM Transactions on Audio, Speech, and Language Processing","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Marafioti et al., 2020] Andres Marafioti, Piotr Majdak, Nicki Holighaus, and Nathanaël Perraudin. Gacela: A generative adversarial context encoder for long audio in- painting of music. IEEE Journal of Selected Topics in Sig- nal Processing, 15(1):120-131, 2020.","title":"Gacela: A generative adversarial context encoder for long audio inpainting of music","authors":"Andres Marafioti, Piotr Majdak, Nicki Holighaus, Nathanaël Perraudin","year":"2020","venue":"IEEE Journal of Selected Topics in Signal Processing","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Min et al., 2023] Lejun Min, Junyan Jiang, Gus Xia, and Jingwei Zhao. Polyffusion: A diffusion model for poly- phonic score generation with internal and external con- trols. In Proceedings of the 24th Conference of the In- ternational Society for Music Information Retrieval, pages 231-238, 2023.","title":"Polyffusion: A diffusion model for polyphonic score generation with internal and external controls","authors":"Min","year":"2023","venue":"Proceedings of the 24th Conference of the International Society for Music Information Retrieval","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Mitsufuji et al., 2022] Yuki Mitsufuji, Giorgio Fabbro, Ste- fan Uhlich, Fabian-Robert Stöter, Alexandre Défossez, Minseok Kim, Woosung Choi, Chin-Yun Yu, and Kin-Wai Cheuk. Music demixing challenge 2021. Frontiers in Sig- nal Processing, 1:808395, 2022.","title":"Music demixing challenge 2021","authors":"Yuki Mitsufuji, Giorgio Fabbro, Stefan Uhlich, Fabian-Robert Stöter, Alexandre Défossez, Minseok Kim, Woosung Choi, Chin-Yun Yu, Kin-Wai Cheuk","year":"2022","venue":"Frontiers in Signal Processing","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Perraudin et al., 2018] Nathanael Perraudin, Nicki Ho- lighaus, Piotr Majdak, and Peter Balazs. Inpainting of long audio segments with similarity graphs. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 26(6):1083-1094, 2018.","title":"Inpainting of long audio segments with similarity graphs","authors":"Perraudin","year":"2018","venue":"IEEE/ACM Transactions on Audio, Speech, and Language Processing","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Raffel et al., 2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485- 5551, 2020.","title":"Exploring the limits of transfer learning with a unified text-to-text transformer","authors":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu","year":"2020","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Schneider et al., 2023] Flavio Schneider, Ojasv Kamal, Zhijing Jin, and Bernhard Schölkopf. Mo\\ˆusai: Text-to- music generation with long-context latent diffusion. arXiv preprint arXiv:2301.11757, 2023.","title":"Mo\\ˆusai: Text-tomusic generation with long-context latent diffusion","authors":"Flavio Schneider, Ojasv Kamal, Zhijing Jin, Bernhard Schölkopf","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Van Den Oord et al., 2017] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.","title":"Neural discrete representation learning","authors":"Oord Van Den","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang and Xia, 2021] Ziyu Wang and Gus Xia. Musebert: Pre-training music representation for music understanding and controllable generation. In ISMIR, pages 722-729, 2021.","title":"Musebert: Pre-training music representation for music understanding and controllable generation","authors":"Wang, Xia ; ] Ziyu Wang, Gus Xia","year":"2021","venue":"ISMIR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wei et al., 2023] Shiqi Wei, Ziyu Wang, Weiguo Gao, and Gus Xia. Controllable music inpainting with mixed-level and disentangled representation. In ICASSP 2023 -2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, 2023.","title":"Controllable music inpainting with mixed-level and disentangled representation","authors":"Shiqi Wei, Ziyu Wang, Weiguo Gao, Gus Xia","year":"2023","venue":"ICASSP 2023 -2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wu et al., 2023a] Shih-Lun Wu, Chris Donahue, Shinji Watanabe, and Nicholas J. Bryan. Music controlnet: Mul- tiple time-varying controls for music generation, 2023.","title":"Music controlnet: Multiple time-varying controls for music generation","authors":"Wu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wu et al., 2023b] Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dub- nov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.","title":"Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation","authors":"Wu","year":"2023","venue":"ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang et al., 2023] Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine- tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199, 2023.","title":"Llama-adapter: Efficient finetuning of language models with zero-init attention","authors":"Zhang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhao and Xia, 2021] Jingwei Zhao and Gus Xia. Accomon- tage: Accompaniment arrangement via phrase selection and style transfer. arXiv preprint arXiv:2108.11213, 2021.","title":"Accomontage: Accompaniment arrangement via phrase selection and style transfer","authors":"Xia ; ] Jingwei Zhao, Gus Zhao, Xia","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhou et al., 2019] Hang Zhou, Ziwei Liu, Xudong Xu, Ping Luo, and Xiaogang Wang. Vision-infused deep audio in- painting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 283-292, 2019.","title":"Vision-infused deep audio inpainting","authors":"Hang Zhou, Ziwei Liu, Xudong Xu, Ping Luo, Xiaogang Wang","year":"2019","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}