{"type":"ScholarlyArticle","title":"Re-creation of Creations: A New Paradigm for Lyric-to-Melody Generation","authors":[{"name":"Ang Lv","slug":"ang-lv","given":"Ang","family":"Lv","affiliation":"Gaoling School of Artificial Intelligence , Renmin University of China"},{"name":"Xu Tan","slug":"xu-tan","given":"Xu","family":"Tan","affiliation":"Ministry of Education"},{"name":"Tao Qin","slug":"tao-qin","given":"Tao","family":"Qin","affiliation":"Ministry of Education"},{"name":"Tie-Yan Liu","slug":"tie-yan-liu","given":"Tie-Yan","family":"Liu","affiliation":"Ministry of Education"},{"name":"Rui Yan","slug":"rui-yan","given":"Rui","family":"Yan","affiliation":"Gaoling School of Artificial Intelligence , Renmin University of China; Engineering Research Center of Next-Generation Intelligent Search and Recommendation ,"}],"abstract":"Current lyric-to-melody generation methods struggle with the lack of paired lyric-melody data to train, and the lack of adherence to composition guidelines, resulting in melodies that do not sound human-composed. To address these issues, we propose a novel paradigm called Re-creation of Creations (ROC) that combines the strengths of both rule-based and neural-based methods. ROC consists of a two-stage generation-retrieval pipeline: the creation and re-creation stages. In the creation stage, we train a melody language model using melody data to generate high-quality music fragments, which are stored in a database indexed by key features. In the re-creation stage, users provide lyrics and a preferred chord progression, and ROC infers melody features for each lyric sentence. By querying the database, we obtain relevant melody fragments that satisfy composition guidelines, and these candidates are filtered, re-ranked, and concatenated based on the guidelines and the melody language model scores. ROC offers two main advantages: it does not require paired lyric-melody data, and it incorporates commonly used composition guidelines, resulting in music that sounds more human-composed with better controllability. Both objective and subjective evaluation results on  English and Chinese lyrics show the effectiveness of ROC.","doi":"10.24963/ijcai.2024/853","url":"https://ijcai.org/proceedings/2024/853","pdf":"https://ijcai.org/proceedings/2024/0853.pdf","markdown":"https://ijcai.org/proceedings/2024/853.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/853","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"AI, Arts & Creativity","track":"AI, Arts & Creativity","number":853,"pages":{"first":7708,"last":7716},"keywords":["Theory and philosophy of arts and creativity in AI systems: Autonomous creative or artistic AI","Application domains: Music and sound"],"videos":[],"references":[{"position":1,"raw":"[Bao et al., 2018] Hangbo Bao, Shaohan Huang, Furu Wei, Lei Cui, Yu Wu, Chuanqi Tan, Songhao Piao, and Ming Zhou. Neural melody composition from lyrics. CoRR, abs/1809.04318, 2018.","title":"Neural melody composition from lyrics","authors":"Bao","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Deng, 2020] Da Deng. Chinese sentiment analysis library, which supports counting the number of different emotional words in the text. https://github.com/hiDaDeng/cnsenti, 2020.","title":"Chinese sentiment analysis library, which supports counting the number of different emotional words in the text","authors":"Da Deng","year":null,"venue":null,"doi":null,"url":"https://github.com/hiDaDeng/cnsenti,2020","internal_url":null},{"position":3,"raw":"[Doll, 2011] Christopher Doll. Rockin' out: Expressive modulation in verse-chorus form. Music Theory Online 17/3 (2011), § 2., 2011.","title":"Rockin' out: Expressive modulation in verse-chorus form","authors":"Christopher Doll","year":"2011","venue":"Music Theory Online","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fell et al., 2018] Michael Fell, Yaroslav Nechaev, Elena Cabrio, and Fabien Gandon. Lyrics segmentation: Tex- tual macrostructure detection using convolutions. In Pro- ceedings of the 27th International Conference on Com- putational Linguistics, pages 2044-2054, Santa Fe, New Mexico, USA, August 2018. Association for Computa- tional Linguistics.","title":"Lyrics segmentation: Textual macrostructure detection using convolutions","authors":"Michael Fell, Yaroslav Nechaev, Elena Cabrio, Fabien Gandon","year":"2018","venue":"Proceedings of the 27th International Conference on Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fukayama et al., 2010] Satoru Fukayama, Kei Nakatsuma, Shinji Sako, Takuya Nishimoto, and Shigeki Sagayama. Automatic song composition from the lyrics exploiting prosody of japanese language. Proceedings of the 7th Sound and Music Computing Conference, SMC 2010, 01 2010.","title":"Automatic song composition from the lyrics exploiting prosody of japanese language","authors":"Satoru Fukayama, Kei Nakatsuma, Shinji Sako, Takuya Nishimoto, Shigeki Sagayama","year":"2010","venue":"Proceedings of the 7th Sound and Music Computing Conference","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gotham et al., 2023] Mark Gotham, Kyle Gullings, Chelsey Hamm, Bryn Hughes, Brian Jarvis, Megan Lavengood, and John Peterson. Open music theory. VERSE-CHORUS FORM, 2023.","title":"Open music theory","authors":"Mark Gotham, Kyle Gullings, Chelsey Hamm, Bryn Hughes, Brian Jarvis, Megan Lavengood, John Peterson","year":"2023","venue":"VERSE-CHORUS FORM","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Guo et al., 2020] Rui Guo, Ivor Simpson, Thor Magnusson, Chris Kiefer, and Dorien Herremans. A variational au- toencoder for music generation controlled by tonal ten- sion. In Joint Conference on AI Music Creativity (CSMC + MuMe), 2020.","title":"A variational autoencoder for music generation controlled by tonal tension","authors":"Rui Guo, Ivor Simpson, Thor Magnusson, Chris Kiefer, Dorien Herremans","year":"2020","venue":"Joint Conference on AI Music Creativity (CSMC + MuMe)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jayaram, 2018] Vivek Jayaram. Pychorus: Python mod- ule for detecting musical choruses. https://github.com/ vivjay30/pychorus, 2018.","title":"Pychorus: Python module for detecting musical choruses","authors":"Vivek Jayaram","year":null,"venue":null,"doi":null,"url":"https://github.com/vivjay30/pychorus,2018","internal_url":null},{"position":9,"raw":"[Ju et al., 2021] Zeqian Ju, Peiling Lu, Xu Tan, Rui Wang, Chen Zhang, Songruoyao Wu, Kejun Zhang, Xiangyang Li, Tao Qin, and Tie-Yan Liu. Telemelody: Lyric- to-melody generation with a template-based two-stage method. CoRR, abs/2109.09617, 2021.","title":"Telemelody: Lyricto-melody generation with a template-based two-stage method","authors":"Ju","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kingma and Ba, 2015] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Inter- national Conference on Learning Representations (ICLR), 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2016] Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-promoting ob- jective function for neural conversation models. In Pro- ceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies, pages 110-119, San Diego, California, June 2016. Association for Computa- tional Linguistics.","title":"A diversity-promoting objective function for neural conversation models","authors":"Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, Bill Dolan","year":"2016","venue":"Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2018] Shen Li, Zhe Zhao, Renfen Hu, Wensi Li, Tao Liu, and Xiaoyong Du. Analogical reasoning on Chinese morphological and semantic relations. In Pro- ceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 138-143, Melbourne, Australia, July 2018. Associ- ation for Computational Linguistics.","title":"Analogical reasoning on Chinese morphological and semantic relations","authors":"Shen Li, Zhe Zhao, Renfen Hu, Wensi Li, Tao Liu, Xiaoyong Du","year":"2018","venue":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2020] Piji Li, Haisong Zhang, Xiaojiang Liu, and Shuming Shi. Rigid formats controlled text generation. In Proceedings of the 58th Annual Meeting of the Associa- tion for Computational Linguistics, pages 742-751, On- line, July 2020. Association for Computational Linguis- tics.","title":"Rigid formats controlled text generation","authors":"Li","year":"2020","venue":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liang et al., 2020] Hongru Liang, Wenqiang Lei, Paul Yaozhu Chan, Zhenglu Yang, Maosong Sun, and Tat-Seng Chua. Pirhdy: Learning pitch-, rhythm-, and dynamics-aware embeddings for symbolic music. CoRR, abs/2010.08091, 2020.","title":"Pirhdy: Learning pitch-, rhythm-, and dynamics-aware embeddings for symbolic music","authors":"Hongru Liang, Wenqiang Lei, Paul Yaozhu Chan, Zhenglu Yang, Maosong Sun, Tat-Seng Chua","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Long et al., 2013] Cheng Long, Raymond Chi wing Wong, and Raymond Ka Wai Sze. T-music: A melody composer based on frequent pattern mining. 2013 IEEE 29th Inter- national Conference on Data Engineering (ICDE), pages 1332-1335, 2013.","title":"T-music: A melody composer based on frequent pattern mining","authors":"Cheng Long, Raymond Chi Wing Wong, Raymond Ka, Wai Sze","year":"2013","venue":"IEEE 29th International Conference on Data Engineering (ICDE)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Loria, 2020] Steven Loria. Textblob v0.16.0 simple, pythonic, text processing-sentiment analysis, part-of- speech tagging, noun phrase extraction, translation, and more. https://github.com/sloria/textblob, 2020.","title":"Textblob v0.16.0 simple, pythonic, text processing-sentiment analysis, part-ofspeech tagging, noun phrase extraction, translation","authors":"Steven Loria","year":null,"venue":null,"doi":null,"url":"https://github.com/sloria/textblob,2020","internal_url":null},{"position":17,"raw":"[Ma et al., 2021] Xichu Ma, Ye Wang, Min-Yen Kan, and Wee Sun Lee. AI-Lyricist: Generating Music and Vocabu- lary Constrained Lyrics, page 1002-1011. Association for Computing Machinery, New York, NY, USA, 2021.","title":"AI-Lyricist: Generating Music and Vocabulary Constrained Lyrics","authors":"Ma","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Magenta, 2020] Magenta. A serializable note sequence representation and utilities. https://github.com/magenta/ note-seq, 2020.","title":"A serializable note sequence representation and utilities","authors":"Magenta","year":null,"venue":null,"doi":null,"url":"https://github.com/magenta/note-seq,2020","internal_url":null},{"position":19,"raw":"[Malmi et al., 2016] Eric Malmi, Pyry Takala, Hannu Toivo- nen, Tapani Raiko, and Aristides Gionis. Dopelearning: A computational approach to rap lyrics generation. In Pro- ceedings of the 22nd ACM SIGKDD International Con- ference on Knowledge Discovery and Data Mining, KDD '16, page 195-204, New York, NY, USA, 2016. Associa- tion for Computing Machinery.","title":"Dopelearning: A computational approach to rap lyrics generation","authors":"Eric Malmi, Pyry Takala, Hannu Toivonen, Tapani Raiko, Aristides Gionis","year":"2016","venue":"Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In Empirical Meth- ods in Natural Language Processing (EMNLP), pages 1532-1543, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher D Manning","year":"2014","venue":"Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Raffel, 2016] Colin Raffel. Learning-based methods for comparing sequences, with applications to audio-to-midi alignment and matching. Columbia University, 2016.","title":"Learning-based methods for comparing sequences, with applications to audio-to-midi alignment and matching","authors":"Colin Raffel","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sheng et al., 2020] Zhonghao Sheng, Kaitao Song, Xu Tan, Yi Ren, Wei Ye, Shikun Zhang, and Tao Qin. Songmass: Automatic song writing with pre-training and alignment constraint. CoRR, abs/2012.05168, 2020.","title":"Songmass: Automatic song writing with pre-training and alignment constraint","authors":"Zhonghao Sheng, Kaitao Song, Xu Tan, Yi Ren, Wei Ye, Shikun Zhang, Tao Qin","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wal- lach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, vol- ume 30.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł Ukasz Kaiser, Illia Polosukhin","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"Curran Associates, Inc., 2017.","title":null,"authors":"Curran Associates, Inc","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Watanabe et al., 2016] Kento Watanabe, Yuichiroh Matsub- ayashi, Naho Orita, Naoaki Okazaki, Kentaro Inui, Satoru Fukayama, Tomoyasu Nakano, Jordan Smith, and Masa- taka Goto. Modeling discourse segments in lyrics using repeated patterns. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguis- tics: Technical Papers, pages 1959-1969, Osaka, Japan, December 2016. The COLING 2016 Organizing Commit- tee.","title":"Modeling discourse segments in lyrics using repeated patterns","authors":"Kento Watanabe, Yuichiroh Matsubayashi, Naho Orita, Naoaki Okazaki, Kentaro Inui, Satoru Fukayama, Tomoyasu Nakano, Jordan Smith, Masataka Goto","year":"2016","venue":"Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wu et al., 2020] Jian Wu, Xiaoguang Liu, Xiaolin Hu, and Jun Zhu. Popmnet: Generating structured pop music melodies using neural networks. Artificial Intelligence, 286:103303, 2020.","title":"Popmnet: Generating structured pop music melodies using neural networks","authors":"Jian Wu, Xiaoguang Liu, Xiaolin Hu, Jun Zhu","year":"2020","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xue et al., 2021] Lanqing Xue, Kaitao Song, Duocai Wu, Xu Tan, Nevin L. Zhang, Tao Qin, Wei-Qiang Zhang, and Tie-Yan Liu. Deeprapper: Neural rap generation with rhyme and rhythm modeling. CoRR, abs/2107.01875, 2021.","title":"Deeprapper: Neural rap generation with rhyme and rhythm modeling","authors":"Lanqing Xue, Kaitao Song, Duocai Wu, Xu Tan, Nevin L Zhang, Tao Qin, Wei-Qiang Zhang, Tie-Yan Liu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yu et al., 2021] Yi Yu, Abhishek Srivastava, and Simon Canales. Conditional lstm-gan for melody generation from lyrics. ACM Trans. Multimedia Comput. Commun. Appl., 17(1), apr 2021.","title":"Conditional lstm-gan for melody generation from lyrics","authors":"Yi Yu, Abhishek Srivastava, Simon Canales","year":"2021","venue":"ACM Trans. Multimedia Comput. Commun. Appl","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zeng et al., 2021] Mingliang Zeng, Xu Tan, Rui Wang, Ze- qian Ju, Tao Qin, and Tie-Yan Liu. MusicBERT: Sym- bolic music understanding with large-scale pre-training. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 791-800, Online, August 2021. Association for Computational Linguistics.","title":"MusicBERT: Symbolic music understanding with large-scale pre-training","authors":"Mingliang Zeng, Xu Tan, Rui Wang, Zeqian Ju, Tao Qin, Tie-Yan Liu","year":"2021","venue":"Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2018] Yizhe Zhang, Michel Galley, Jianfeng Gao, Zhe Gan, Xiujun Li, Chris Brockett, and Bill Dolan. Generating informative and diverse conversa- tional responses via adversarial information maximization. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, page 1815-1825, Red Hook, NY, USA, 2018.","title":"Generating informative and diverse conversational responses via adversarial information maximization","authors":"Zhang","year":"2018","venue":"Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Curran Asso- ciates Inc. Zhu et al., 2018] Hongyuan Zhu, Qi Liu, Nicholas Jing Yuan, Chuan Qin, Jiawei Li, Kun Zhang, Guang Zhou, Furu Wei, Yuanchun Xu, and Enhong Chen. Xiaoice band: A melody and arrangement generation framework for pop music. In Proceedings of the 24th ACM SIGKDD Inter- national Conference on Knowledge Discovery and Data Mining, page 2837-2846, New York, NY, USA, 2018. As- sociation for Computing Machinery.","title":"Xiaoice band: A melody and arrangement generation framework for pop music","authors":"Hongyuan Zhu, Qi Liu, Nicholas Jing Yuan, Chuan Qin, Jiawei Li, Kun Zhang, Guang Zhou, Furu Wei, Yuanchun Xu, Enhong Chen","year":"2018","venue":"Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}