{"type":"ScholarlyArticle","title":"Humming2Music: Being A Composer As Long As You Can Humming","authors":[{"name":"Yao Qiu","slug":"yao-qiu","given":"Yao","family":"Qiu","affiliation":""},{"name":"Jinchao Zhang","slug":"jinchao-zhang","given":"Jinchao","family":"Zhang","affiliation":""},{"name":"Huiying Ren","slug":"huiying-ren","given":"Huiying","family":"Ren","affiliation":""},{"name":"Yong Shan","slug":"yong-shan","given":"Yong","family":"Shan","affiliation":""},{"name":"Jie Zhou","slug":"jie-zhou","given":"Jie","family":"Zhou","affiliation":""}],"abstract":"Creating a piece of music is difficult for people who have never been trained to compose. We present an automatic music generation system to lower the threshold of creating music. The system takes the user's humming as input and creates full music based on the humming melody.  The system consists of five modules: 1) humming transcription, 2) melody generation, 3) broken chord generation, 4) accompaniment generation, and 5) audio synthesis. The first module transcribes the user's humming audio to a score, and then the melody generation module composes a complete melody based on the user's humming melody. After that, the third module will generate a broken chord track to accompany the full melody, and the fourth module will create more accompanying tracks. Finally, the audio synthesis module mixes all the tracks to generate the music. Through the user experiment, our system can generate high-quality music with natural expression based on the user's humming input.","doi":"10.24963/ijcai.2023/840","url":"https://ijcai.org/proceedings/2023/840","pdf":"https://ijcai.org/proceedings/2023/0840.pdf","markdown":"https://ijcai.org/proceedings/2023/840.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/840","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Demo Track","track":"Demo Track","number":840,"pages":{"first":7163,"last":7166},"keywords":["Multidisciplinary Topics and Applications: MDA: Other","Multidisciplinary Topics and Applications: MDA: Arts and creativity"],"videos":[],"references":[{"position":1,"raw":"[Agostinelli et al., 2023] Andrea Agostinelli, Timo I Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Cail- lon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text. arXiv preprint arXiv:2301.11325, 2023.","title":"Musiclm: Generating music from text","authors":"Andrea Agostinelli, I Timo, Zalán Denk, Jesse Borsos, Mauro Engel, Antoine Verzetti, Qingqing Caillon, Aren Huang, Adam Jansen, Marco Roberts, Tagliasacchi","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Choi et al., 2021] Kyoyun Choi, Jonggwon Park, Wan Heo, Sungwook Jeon, and Jonghun Park. Chord conditioned melody generation with transformer based decoders. IEEE Access, 9:42071-42080, 2021.","title":"Chord conditioned melody generation with transformer based decoders","authors":"Choi","year":"2021","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dhariwal et al., 2020] Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint arXiv:2005.00341, 2020.","title":"Jukebox: A generative model for music","authors":"Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, Ilya Sutskever","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Hawthorne et al., 2017] Curtis Hawthorne, Erich Elsen, Jialin Song, Adam Roberts, Ian Simon, Colin Raffel, Jesse Engel, Sageev Oore, and Douglas Eck. Onsets and frames: Dual-objective piano transcription. arXiv preprint arXiv:1710.11153, 2017.","title":"Onsets and frames: Dual-objective piano transcription","authors":"Curtis Hawthorne, Erich Elsen, Jialin Song, Adam Roberts, Ian Simon, Colin Raffel, Jesse Engel, Sageev Oore, Douglas Eck","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hsiao et al., 2021] Wen-Yi Hsiao, Jen-Yu Liu, Yin-Cheng Yeh, and Yi-Hsuan Yang. Compound word transformer: Learning to compose full-song music over dynamic di- rected hypergraphs. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 35, pages 178-186, 2021.","title":"Compound word transformer: Learning to compose full-song music over dynamic directed hypergraphs","authors":"Wen-Yi Hsiao, Jen-Yu Liu, Yin-Cheng Yeh, Yi-Hsuan Yang","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Huang and Yang, 2020] Yu-Siang Huang and Yi-Hsuan Yang. Pop music transformer: Beat-based modeling and generation of expressive pop piano compositions. In Pro- ceedings of the 28th ACM International Conference on Multimedia, pages 1180-1188, 2020.","title":"Pop music transformer: Beat-based modeling and generation of expressive pop piano compositions","authors":"Yu-Siang Huang, Yi-Hsuan Yang","year":"2020","venue":"Proceedings of the 28th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Huang et al., 2018] Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer. arXiv preprint arXiv:1809.04281, 2018.","title":"Music transformer","authors":"Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, Douglas Eck","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ju et al., 2021] Zeqian Ju, Peiling Lu, Xu Tan, Rui Wang, Chen Zhang, Songruoyao Wu, Kejun Zhang, Xiangyang Li, Tao Qin, and Tie-Yan Liu. Telemelody: Lyric- to-melody generation with a template-based two-stage method. arXiv preprint arXiv:2109.09617, 2021.","title":"Telemelody: Lyricto-melody generation with a template-based two-stage method","authors":"Ju","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kelz et al., 2016] Rainer Kelz, Matthias Dorfer, Filip Ko- rzeniowski, Sebastian Böck, Andreas Arzt, and Gerhard Widmer. On the potential of simple framewise approaches to piano transcription. arXiv preprint arXiv:1612.05153, 2016.","title":"On the potential of simple framewise approaches to piano transcription","authors":"Rainer Kelz, Matthias Dorfer, Filip Korzeniowski, Sebastian Böck, Andreas Arzt, Gerhard Widmer","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Liu et al., 2022] Jinglin Liu, Chengxi Li, Yi Ren, Feiyang Chen, and Zhou Zhao. Diffsinger: Singing voice synthe- sis via shallow diffusion mechanism. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11020-11028, 2022.","title":"Diffsinger: Singing voice synthesis via shallow diffusion mechanism","authors":"Jinglin Liu, Chengxi Li, Yi Ren, Feiyang Chen, Zhou Zhao","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[McClung, 2001] Alan C McClung. Sight-singing systems: Current practice and survey of all-state choristers. Update: Applications of Research in Music Education, 20(1):3-8, 2001.","title":"Sight-singing systems: Current practice and survey of all-state choristers","authors":"Alan C Mcclung","year":"2001","venue":"Update: Applications of Research in Music Education","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Oord et al., 2016] Aaron van den Oord, Sander Diele- man, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw au- dio. arXiv preprint arXiv:1609.03499, 2016.","title":"Wavenet: A generative model for raw audio","authors":"Oord","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Payne, 2019] Christine Payne. Musenet. OpenAI Blog, 3, 2019.","title":"OpenAI Blog","authors":"Christine Payne, Musenet","year":"2019","venue":"OpenAI Blog","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Sigtia et al., 2016] Siddharth Sigtia, Emmanouil Benetos, and Simon Dixon. An end-to-end neural network for polyphonic piano music transcription. IEEE/ACM Trans- actions on Audio, Speech, and Language Processing, 24(5):927-939, 2016.","title":"An end-to-end neural network for polyphonic piano music transcription","authors":"Siddharth Sigtia, Emmanouil Benetos, Simon Dixon","year":"2016","venue":"IEEE/ACM Transactions on Audio, Speech, and Language Processing","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Simonetta et al., 2018] Federico Simonetta, Filippo Carnovalini, Nicola Orio, and Antonio Rodà. Symbolic music similarity through a graph-based representation. In Proceedings of the Audio Mostly 2018 on Sound in Immersion and Emotion, pages 1-7. 2018.","title":"Symbolic music similarity through a graph-based representation","authors":"Federico Simonetta, Filippo Carnovalini, Nicola Orio, Antonio Rodà","year":"2018","venue":"Proceedings of the Audio Mostly 2018 on Sound in Immersion and Emotion","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Wang and Jang, 2021] Jun-You Wang and Jyh-Shing Roger Jang. On the preparation and validation of a large-scale dataset of singing transcription. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 276-280. IEEE, 2021.","title":"On the preparation and validation of a large-scale dataset of singing transcription","authors":"Wang, Jang ; Jun-You Wang, Jyh-Shing Roger Jang","year":"2021","venue":"ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2020] Ziyu Wang, Ke Chen, Junyan Jiang, Yiyi Zhang, Maoran Xu, Shuqi Dai, Xianbin Gu, and Gus Xia. Pop909: A pop-song dataset for music arrangement gen- eration. arXiv preprint arXiv:2008.07142, 2020.","title":"Pop909: A pop-song dataset for music arrangement generation","authors":"Ziyu Wang, Ke Chen, Junyan Jiang, Yiyi Zhang, Maoran Xu, Shuqi Dai, Xianbin Gu, Gus Xia","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2022] Xianke Wang, Wei Xu, Weiming Yang, and Wenqing Cheng. Musicyolo: A sight-singing on- set/offset detection framework based on object detection instead of spectrum frames. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 396-400. IEEE, 2022.","title":"Musicyolo: A sight-singing onset/offset detection framework based on object detection instead of spectrum frames","authors":"Wang","year":"2022","venue":"ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wu et al., 2019] Jian Wu, Changran Hu, Yulong Wang, Xi- aolin Hu, and Jun Zhu. A hierarchical recurrent neural net- work for symbolic melody generation. IEEE transactions on cybernetics, 50(6):2749-2757, 2019.","title":"A hierarchical recurrent neural network for symbolic melody generation","authors":"Jian Wu, Changran Hu, Yulong Wang, Xiaolin Hu, Jun Zhu","year":"2019","venue":"IEEE transactions on cybernetics","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"Zhang et al., ] Lichao Zhang, Ruiqi Li, Shoutong Wang, Liqun Deng, Jinglin Liu, Yi Ren, Jinzheng He, Rongjie Huang, Jieming Zhu, Xiao Chen, et al. M4singer: A multi- style, multi-singer and musical score provided mandarin singing corpus. In Thirty-sixth Conference on Neural In- formation Processing Systems Datasets and Benchmarks Track.","title":"M4singer: A multistyle, multi-singer and musical score provided mandarin singing corpus","authors":"Zhang","year":null,"venue":"Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhang et al., 2022] Xueyao Zhang, Jinchao Zhang, Yao Qiu, Li Wang, and Jie Zhou. Structure-enhanced pop mu- sic generation via harmony-aware learning. In Proceed- ings of the 30th ACM International Conference on Multi- media, pages 1204-1213, 2022.","title":"Structure-enhanced pop music generation via harmony-aware learning","authors":"Zhang","year":"2022","venue":"Proceedings of the 30th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}