{"type":"ScholarlyArticle","title":"LingGe: An Automatic Ancient Chinese Poem-to-Song Generation System","authors":[{"name":"Yong Shan","slug":"yong-shan","given":"Yong","family":"Shan","affiliation":""},{"name":"Jinchao Zhang","slug":"jinchao-zhang","given":"Jinchao","family":"Zhang","affiliation":""},{"name":"Huiying Ren","slug":"huiying-ren","given":"Huiying","family":"Ren","affiliation":""},{"name":"Yao Qiu","slug":"yao-qiu","given":"Yao","family":"Qiu","affiliation":""},{"name":"Jie Zhou","slug":"jie-zhou","given":"Jie","family":"Zhou","affiliation":""}],"abstract":"This paper presents a novel system, named LingGe (\"伶歌\" in Chinese), to generate songs for ancient Chinese poems automatically. LingGe takes the poem as the lyric, composes music conditioned on the lyric, and finally outputs a full song including the singing and the accompaniment. It consists of four modules: rhythm recognition, melody generation, accompaniment generation, and audio synthesis. Firstly, the rhythm recognition module analyzes the song structure and rhythm according to the poem. Secondly, the melody generation module assembles the rhythm into the template and then generates the melody. Thirdly, the accompaniment generation module predicts the accompaniment in harmony with the melody. Finally, the audio synthesis module generates singing and accompaniment audio and then mixes them to obtain songs. The results show that LingGe can generate high-quality and expressive songs for ancient Chinese poems, both in harmony and rhythm.","doi":"10.24963/ijcai.2023/842","url":"https://ijcai.org/proceedings/2023/842","pdf":"https://ijcai.org/proceedings/2023/0842.pdf","markdown":"https://ijcai.org/proceedings/2023/842.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/842","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Demo Track","track":"Demo Track","number":842,"pages":{"first":7171,"last":7174},"keywords":["Multidisciplinary Topics and Applications: MDA: Arts and creativity","Multidisciplinary Topics and Applications: MDA: Other"],"videos":[],"references":[{"position":1,"raw":"[Bao et al., 2018] Hangbo Bao, Shaohan Huang, Furu Wei, Lei Cui, Yuehua Wu, Chuanqi Tan, Songhao Piao, and M. Zhou. Neural melody composition from lyrics. In Nat- ural Language Processing and Chinese Computing, 2018.","title":"Neural melody composition from lyrics","authors":"Bao","year":"2018","venue":"Natural Language Processing and Chinese Computing","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Huang et al., 2021] Rongjie Huang, Feiyang Chen, Yi Ren, Jinglin Liu, Chenye Cui, and Zhou Zhao. Multi-singer: Fast multi-singer singing voice vocoder with a large-scale corpus. Proceedings of the 29th ACM International Con- ference on Multimedia, 2021.","title":"Multi-singer: Fast multi-singer singing voice vocoder with a large-scale corpus","authors":"Huang","year":"2021","venue":"Proceedings of the 29th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ju et al., 2021] Zeqian Ju, Peiling Lu, Xu Tan, Rui Wang, Chen Zhang, Songruoyao Wu, Ke jun Zhang, Xiang- Yang Li, Tao Qin, and Tie-Yan Liu. Telemelody: Lyric- to-melody generation with a template-based two-stage method. In Conference on Empirical Methods in Natural Language Processing, 2021.","title":"Telemelody: Lyricto-melody generation with a template-based two-stage method","authors":"Ju","year":"2021","venue":"Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Kong et al., 2020] Jungil Kong, Jaehyeon Kim, and Jaeky- oung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33:17022-17033, 2020.","title":"Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis","authors":"Kong","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Lee et al., 2019] Hsin-Pei Lee, Jhih-Sheng Fang, and Wei- Yun Ma. icomposer: An automatic songwriting system for chinese popular music. In North American Chapter of the Association for Computational Linguistics, 2019.","title":"icomposer: An automatic songwriting system for chinese popular music","authors":"Hsin-Pei Lee, Jhih-Sheng Fang, Wei-Yun Ma","year":"2019","venue":"North American Chapter of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Liu et al., 2021] Jinglin Liu, Chengxi Li, Yi Ren, Feiyang Chen, and Zhou Zhao. Diffsinger: Singing voice synthesis via shallow diffusion mechanism. In AAAI Conference on Artificial Intelligence, 2021.","title":"Diffsinger: Singing voice synthesis via shallow diffusion mechanism","authors":"Jinglin Liu, Chengxi Li, Yi Ren, Feiyang Chen, Zhou Zhao","year":"2021","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Lv et al., 2022] Ang Lv, Xu Tan, Tao Qin, Tie-Yan Liu, and Rui Yan. Re-creation of creations: A new paradigm for lyric-to-melody generation. ArXiv, abs/2208.05697, 2022.","title":"Re-creation of creations: A new paradigm for lyric-to-melody generation","authors":"Ang Lv, Xu Tan, Tao Qin, Tie-Yan Liu, Rui Yan","year":"2022","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2024/853"},{"position":8,"raw":"[Newmarch and Newmarch, 2017] Jan Newmarch and Jan Newmarch. Fluidsynth. Linux Sound Programming, pages 351-353, 2017.","title":"Jan Newmarch and Jan Newmarch. Fluidsynth. Linux Sound Programming","authors":"Newmarch Newmarch","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Qian et al., 2022] Tao Qian, Jiatong Shi, Shuai Guo, Peter Wu, and Qin Jin. Training strategies for automatic song writing: A unified framework perspective. ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4738-4742, 2022.","title":"Training strategies for automatic song writing: A unified framework perspective","authors":"Tao Qian, Jiatong Shi, Shuai Guo, Peter Wu, Qin Jin","year":"2022","venue":"ICASSP 2022 -2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Sheng et al., 2021] Zhonghao Sheng, Kaitao Song, Xu Tan, Yi Ren, Wei Ye, Shikun Zhang, and Tao Qin. Songmass: Automatic song writing with pre-training and alignment constraint. In Proceedings of the AAAI Conference on Ar- tificial Intelligence, volume 35, pages 13798-13805, 2021.","title":"Songmass: Automatic song writing with pre-training and alignment constraint","authors":"Zhonghao Sheng, Kaitao Song, Xu Tan, Yi Ren, Wei Ye, Shikun Zhang, Tao Qin","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Wang et al., 2020] Ziyu Wang, K. Chen, Junyan Jiang, Yiyi Zhang, Maoran Xu, Shuqi Dai, Xianbin Gu, and Gus Xia. Pop909: A pop-song dataset for music arrangement gen- eration. In International Society for Music Information Retrieval Conference, 2020.","title":"Pop909: A pop-song dataset for music arrangement generation","authors":"Ziyu Wang, K Chen, Junyan Jiang, Yiyi Zhang, Maoran Xu, Shuqi Dai, Xianbin Gu, Gus Xia","year":"2020","venue":"International Society for Music Information Retrieval Conference","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Yu and Canales, 2019] Yi Yu and Simon Canales. Condi- tional lstm-gan for melody generation from lyrics. ACM Transactions on Multimedia Computing, Communica- tions, and Applications (TOMM), 17:1 -20, 2019.","title":"Conditional lstm-gan for melody generation from lyrics","authors":"Yu, Canales, Yi Yu, Simon Canales","year":"2019","venue":"ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}