{"type":"ScholarlyArticle","title":"Demonstration of PerformanceNet: A Convolutional Neural Network Model for Score-to-Audio Music Generation","authors":[{"name":"Yu-Hua Chen","slug":"yu-hua-chen","given":"Yu-Hua","family":"Chen","affiliation":"Research Center for IT Innovation , Academia Sinica , Taiwan"},{"name":"Bryan Wang","slug":"bryan-wang","given":"Bryan","family":"Wang","affiliation":"Research Center for IT Innovation , Academia Sinica , Taiwan"},{"name":"Yi-Hsuan Yang","slug":"yi-hsuan-yang","given":"Yi-Hsuan","family":"Yang","affiliation":"Research Center for IT Innovation , Academia Sinica , Taiwan"}],"abstract":"We present in this paper PerformacnceNet, a neural network model we proposed recently to achieve\r\nscore-to-audio music generation. The model learns\r\nto convert a music piece from the symbolic domain\r\nto the audio domain, assigning performance-level\r\nattributes such as changes in velocity automatically\r\nto the music and then synthesizing the audio. The\r\nmodel is therefore not just a neural audio synthesizer, but an AI performer that learns to interpret a\r\nmusical score in its own way. The code and sample outputs of the model can be found online at\r\nhttps://github.com/bwang514/PerformanceNet.","doi":"10.24963/ijcai.2019/938","url":"https://ijcai.org/proceedings/2019/938","pdf":"https://ijcai.org/proceedings/2019/0938.pdf","markdown":"https://ijcai.org/proceedings/2019/938.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/938","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Demos","track":"Demos","number":938,"pages":{"first":6506,"last":6508},"keywords":["AI: Human-Computer Interactive Systems","AI: Machine Learning"],"videos":[],"references":[{"position":1,"raw":"[Brunner et al., 2018] Gino Brunner, Andres Konrad, Yuyi Wang, and Roger Wattenhofer. MIDI-VAE: Modeling dy- namics and instrumentation of music with applications to style transfer. In Proc. Int. Soc. Music Information Re- trieval Conf., pages 23-27, 2018.","title":"MIDI-VAE: Modeling dynamics and instrumentation of music with applications to style transfer","authors":"Gino Brunner, Andres Konrad, Yuyi Wang, Roger Wattenhofer","year":"2018","venue":"Proc. Int. Soc. Music Information Retrieval Conf","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2015] Yuan-Ping Chen, Li Su, and Yi-Hsuan Yang. Electric guitar playing technique detection in real- world recordings based on f0 sequence pattern recogni- tion. In Proc. Int. Soc. Music Information Retrieval Conf., pages 708-714, 2015.","title":"Electric guitar playing technique detection in realworld recordings based on f0 sequence pattern recognition","authors":"Yuan-Ping Chen, Li Su, Yi-Hsuan Yang","year":"2015","venue":"Proc. Int. Soc. Music Information Retrieval Conf","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dong et al., 2018a] Hao-Wen Dong, Wen-Yi Hsiao, Li- Chia Yang, and Yi-Hsuan Yang. MuseGAN: Symbolic- domain music generation and accompaniment with multi- track sequential generative adversarial networks. In Proc. AAAI Conf. Artificial Intelligence, 2018.","title":"MuseGAN: Symbolicdomain music generation and accompaniment with multi-track sequential generative adversarial networks","authors":"Dong","year":"2018","venue":"Proc. AAAI Conf. Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dong et al., 2018b] Hao-Wen Dong, Wen-Yi Hsiao, and Yi- Hsuan Yang. Pypianoroll: Open source Python package for handling multitrack pianoroll. In Proc. Int. Soc. Music Information Retrieval Conf., 2018. Late-breaking paper; [Online] https://github.com/salu133445/pypianoroll.","title":"Pypianoroll: Open source Python package for handling multitrack pianoroll","authors":"Dong","year":"2018","venue":"Proc. Int. Soc. Music Information Retrieval Conf","doi":null,"url":"https://github.com/salu133445/pypianoroll","internal_url":null},{"position":5,"raw":"[Engel et al., 2017] Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Douglas Eck, Karen Si- monyan, and Mohammad Norouzi. Neural audio synthe- sis of musical notes with WaveNet autoencoders. arXiv preprint arXiv:1704.01279, 2017.","title":"Neural audio synthesis of musical notes with WaveNet autoencoders","authors":"Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Douglas Eck, Karen Simonyan, Mohammad Norouzi","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Engel et al., 2019] Jesse Engel, Kumar Krishna Agrawal, Shuo Chen, Ishaan Gulrajani, Chris Donahue, and Adam Roberts. GANSynth: Adversarial neural audio synthesis. In Proc. Int. Conf. Learning Representations, 2019.","title":"GANSynth: Adversarial neural audio synthesis","authors":"Jesse Engel, Krishna Kumar, Shuo Agrawal, Ishaan Chen, Chris Gulrajani, Adam Donahue, Roberts","year":"2019","venue":"Proc. Int. Conf. Learning Representations","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gatys et al., 2016] Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, pages 2414-2423, 2016.","title":"Image style transfer using convolutional neural networks","authors":"Leon A Gatys, Alexander S Ecker, Matthias Bethge","year":"2016","venue":"Proc. IEEE Conf. Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang et al., 2018] Sicong Huang, Qiyang Li, Cem Anil, Xuchan Bao, Sageev Oore, and Roger B. Grosse. Tim- bretron: A wavenet(cyclegan(cqt(audio))) pipeline for mu- sical timbre transfer. CoRR, abs/1811.09620, 2018.","title":"Timbretron: A wavenet(cyclegan(cqt(audio))) pipeline for musical timbre transfer","authors":"Huang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hung et al., 2019] Yun-Ning Hung, I-Tung Chiang, Yi-An Chen, and Yi-Hsuan Yang. Musical composition style transfer via disentangled timbre representations. In Proc. Int. Joint Conf. Artificial Intelligence, 2019.","title":"Musical composition style transfer via disentangled timbre representations","authors":"Yun-Ning Hung, I-Tung Chiang, Yi-An Chen, Yi-Hsuan Yang","year":"2019","venue":"Proc. Int. Joint Conf. Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2019/652"},{"position":10,"raw":"[Kim et al., 2019] Jong Wook Kim, Rachel Bittner, Aparna Kumar, and Juan Pablo Bello. Neural music synthesis for flexible timbre control. In Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing, pages 176-180, 2019.","title":"Neural music synthesis for flexible timbre control","authors":"Jong Wook, Kim, Rachel Bittner, Aparna Kumar, Juan Pablo Bello","year":"2019","venue":"Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ledig et al., 2017] Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. Photo-realistic single im- age super-resolution using a generative adversarial net- work. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, pages 4681-4690, 2017.","title":"Photo-realistic single image super-resolution using a generative adversarial network","authors":"Ledig","year":"2017","venue":"Proc. IEEE Conf. Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2015] Pei-Ching Li, Li Su, Yi-Hsuan Yang, and Alvin W. Y. Su. Analysis of expressive musical terms in vi- olin using score-informed and expression-based audio fea- tures. In Proc. Int. Soc. Music Information Retrieval Conf., pages 809-815, 2015.","title":"Analysis of expressive musical terms in violin using score-informed and expression-based audio features","authors":"Pei-Ching Li, Li Su, Yi-Hsuan Yang, Alvin W Y Su","year":"2015","venue":"Proc. Int. Soc. Music Information Retrieval Conf","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Marafioti et al., 2019] Andrés Marafioti, Nicki Holighaus, Nathanaël Perraudin, and Piotr Majdak. Adversarial gen- eration of time-frequency features with application in au- dio synthesis. CoRR, abs/1902.04072, 2019.","title":"Adversarial generation of time-frequency features with application in audio synthesis","authors":"Andrés Marafioti, Nicki Holighaus, Nathanaël Perraudin, Piotr Majdak","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Shih et al., 2017] Chi-Ching Shih, Pei-Ching Li, Yi-Ju Lin, Alvin W. Y. Su, Li Su, and Yi-Hsuan Yang. Analysis of expressive musical terms in violin using score-informed and expression-based audio features. In Proc. Int. Conf. Digital Audio Effects, 2017.","title":"Analysis of expressive musical terms in violin using score-informed and expression-based audio features","authors":"Shih","year":"2017","venue":"Proc. Int. Conf. Digital Audio Effects","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Simon et al., 2018] Ian Simon, Adam Roberts, Colin Raf- fel, Jesse Engel, Curtis Hawthorne, and Douglas Eck. Learning a latent space of multitrack measures. In Proc. Int. Soc. Music Information Retrieval Conf., 2018.","title":"Learning a latent space of multitrack measures","authors":"Ian Simon, Adam Roberts, Colin Raffel, Jesse Engel, Curtis Hawthorne, Douglas Eck","year":"2018","venue":"Proc. Int. Soc. Music Information Retrieval Conf","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Su et al., 2019] Ting-Wei Su, Yuan-Ping Chen, Li Su, and Yi-Hsuan Yang. TENT: Technique-embedded note track- ing for real-world guitar solo recordings. Transactions of the International Society for Music Information Retrieval, 2019.","title":"TENT: Technique-embedded note tracking for real-world guitar solo recordings","authors":"Ting-Wei Su, Yuan-Ping Chen, Li Su, Yi-Hsuan Yang","year":"2019","venue":"Transactions of the International Society for Music Information Retrieval","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Wang and Yang, 2019] Bryan Wang and Yi-Hsuan Yang. PerformanceNet: Score-to-audio music generation with multi-band convolutional residual network. In Proc. AAAI Conf. Artificial Intelligence, 2019.","title":"PerformanceNet: Score-to-audio music generation with multi-band convolutional residual network","authors":"Bryan Wang, Yi-Hsuan Yang","year":"2019","venue":"Proc. AAAI Conf. Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Yang et al., 2017] Li-Chia Yang, Szu-Yu Chou, and Yi- Hsuan Yang. MidiNet: A convolutional generative adver- sarial network for symbolic-domain music generation. In Proc. Int. Soc. Music Information Retrieval Conf., 2017.","title":"MidiNet: A convolutional generative adversarial network for symbolic-domain music generation","authors":"Li-Chia Yang, Szu-Yu Chou, Yi-Hsuan Yang","year":"2017","venue":"Proc. Int. Soc. Music Information Retrieval Conf","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}