{"type":"ScholarlyArticle","title":"Graph-based Polyphonic Multitrack Music Generation","authors":[{"name":"Emanuele Cosenza","slug":"emanuele-cosenza","given":"Emanuele","family":"Cosenza","affiliation":"University of Pisa"},{"name":"Andrea Valenti","slug":"andrea-valenti","given":"Andrea","family":"Valenti","affiliation":"University of Pisa"},{"name":"Davide Bacciu","slug":"davide-bacciu","given":"Davide","family":"Bacciu","affiliation":"University of Pisa"}],"abstract":"Graphs can be leveraged to model polyphonic multitrack symbolic music, where notes, chords and entire sections may be linked at different levels of the musical hierarchy by tonal and rhythmic relationships. Nonetheless, there is a lack of works that consider graph representations in the context of deep learning systems for music generation. This paper bridges this gap by introducing a novel graph representation for music and a deep Variational Autoencoder that generates the structure and the content of musical graphs separately, one after the other, with a hierarchical architecture that matches the structural priors of music. By separating the structure and content of musical graphs, it is possible to condition generation by specifying which instruments are played at certain times. This opens the door to a new form of human-computer interaction in the context of music co-creation. After training the model on existing MIDI datasets, the experiments show that the model is able to generate appealing short and long musical sequences and to realistically interpolate between them, producing music that is tonally and rhythmically consistent. Finally, the visualization of the embeddings shows that the model is able to organize its latent space in accordance with known musical concepts.","doi":"10.24963/ijcai.2023/643","url":"https://ijcai.org/proceedings/2023/643","pdf":"https://ijcai.org/proceedings/2023/0643.pdf","markdown":"https://ijcai.org/proceedings/2023/643.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/643","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"AI and Arts","track":"AI and Arts","number":643,"pages":{"first":5797,"last":5805},"keywords":["Theory and philosophy of arts and creativity in AI systems: Social (multi-agent) creativity and human-computer co-creation","Theory and philosophy of arts and creativity in AI systems: Autonomous creative or artistic AI","Methods and resources: Machine learning, deep learning, neural models, reinforcement learning","Theory and philosophy of arts and creativity in AI systems: Support of human creativity","Application domains: Music and sound"],"videos":[],"references":[{"position":1,"raw":"[Agostinelli et al., 2023] Andrea Agostinelli, Timo I Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Cail- lon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text. arXiv preprint arXiv:2301.11325, 2023.","title":"Musiclm: Generating music from text","authors":"Andrea Agostinelli, I Timo, Zalán Denk, Jesse Borsos, Mauro Engel, Antoine Verzetti, Qingqing Caillon, Aren Huang, Adam Jansen, Marco Roberts, Tagliasacchi","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bacciu et al., 2020] Davide Bacciu, Federico Errica, Alessio Micheli, and Marco Podda. A gentle intro- duction to deep learning for graphs. Neural Networks, 129:203-221, 2020.","title":"A gentle introduction to deep learning for graphs","authors":"Davide Bacciu, Federico Errica, Alessio Micheli, Marco Podda","year":"2020","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brown et al., 2020] Tom Brown, Benjamin Mann, Nick Ry- der, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing sys- tems, 33:1877-1901, 2020.","title":"Language models are few-shot learners","authors":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brunner et al., 2017] Gino Brunner, Yuyi Wang, Roger Wattenhofer, and Jonas Wiesendanger. Jambot: Music theory aware chord based generation of polyphonic music with lstms. In 2017 IEEE 29th International Conference on Tools with Artificial Intelligence (ICTAI), pages 519- 526. IEEE, 2017.","title":"Jambot: Music theory aware chord based generation of polyphonic music with lstms","authors":"Gino Brunner, Yuyi Wang, Roger Wattenhofer, Jonas Wiesendanger","year":"2017","venue":"IEEE 29th International Conference on Tools with Artificial Intelligence (ICTAI)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Brunner et al., 2018] Gino Brunner, Andres Konrad, Yuyi Wang, and Roger Wattenhofer. Midi-vae: Modeling dy- namics and instrumentation of music with applications to style transfer. arXiv preprint arXiv:1809.07600, 2018.","title":"Midi-vae: Modeling dynamics and instrumentation of music with applications to style transfer","authors":"Gino Brunner, Andres Konrad, Yuyi Wang, Roger Wattenhofer","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chu et al., 2016] Hang Chu, Raquel Urtasun, and Sanja Fi- dler. Song from pi: A musically plausible network for pop music generation. arXiv preprint arXiv:1611.03477, 2016.","title":"Song from pi: A musically plausible network for pop music generation","authors":"Chu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chuan and Herremans, 2018] Ching-Hua Chuan and Dorien Herremans. Modeling temporal tonal relations in polyphonic music through deep networks with a novel image-based representation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.","title":"Modeling temporal tonal relations in polyphonic music through deep networks with a novel image-based representation","authors":"Herremans ; Ching-Hua Chuan, Dorien Chuan, Herremans","year":"2018","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dong et al., 2018] Hao-Wen Dong, Wen-Yi Hsiao, Li-Chia Yang, and Yi-Hsuan Yang. Musegan: Multi-track sequen- tial generative adversarial networks for symbolic music generation and accompaniment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.","title":"Musegan: Multi-track sequential generative adversarial networks for symbolic music generation and accompaniment","authors":"Hao-Wen, Wen-Yi Dong, Li-Chia Hsiao, Yi-Hsuan Yang, Yang","year":"2018","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ens and Pasquier, 2021] Jeffrey Ens and Philippe Pasquier. Building the metamidi dataset: Linking symbolic and au- dio musical data. In ISMIR, pages 182-188, 2021.","title":"Building the metamidi dataset: Linking symbolic and audio musical data","authors":"Pasquier Ens, Jeffrey Ens, Philippe Pasquier","year":"2021","venue":"ISMIR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ferretti, 2017] Stefano Ferretti. On the modeling of mu- sical solos as complex networks. Information Sciences, 375:271-295, 2017.","title":"On the modeling of musical solos as complex networks","authors":"Stefano Ferretti","year":"2017","venue":"Information Sciences","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ferretti, 2018] Stefano Ferretti. On the complex network structure of musical pieces: analysis of some use cases from different music genres. Multimedia Tools and Appli- cations, 77(13):16003-16029, 2018.","title":"On the complex network structure of musical pieces: analysis of some use cases from different music genres","authors":"Stefano Ferretti","year":"2018","venue":"Multimedia Tools and Applications","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gilmer et al., 2017] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neu- ral message passing for quantum chemistry. In Interna- tional conference on machine learning, pages 1263-1272.","title":"Neural message passing for quantum chemistry","authors":"Justin Gilmer, S Samuel, Patrick F Schoenholz, Oriol Riley, George E Vinyals, Dahl","year":null,"venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"PMLR, 2017.","title":null,"authors":"Pmlr","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. Advances in neural information processing systems, 27, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Goodfellow et al., 2016] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http://www.deeplearningbook.org.","title":"Deep Learning","authors":"Ian Goodfellow, Yoshua Bengio, Aaron Courville","year":"2016","venue":null,"doi":null,"url":"http://www.deeplearningbook.org","internal_url":null},{"position":16,"raw":"[Higgins et al., 2016] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained varia- tional framework. 2016. Huang et al., 2018]","title":"beta-vae: Learning basic visual concepts with a constrained variational framework","authors":"Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, Alexander Lerchner","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer. arXiv preprint arXiv:1809.04281, 2018.","title":"Music transformer","authors":"Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, Douglas Eck","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Huang et al., 2019] Cheng-Zhi Anna Huang, Tim Cooij- mans, Adam Roberts, Aaron Courville, and Douglas Eck. Counterpoint by convolution. arXiv preprint arXiv:1903.07227, 2019.","title":"Counterpoint by convolution","authors":"Cheng-Zhi Anna Huang, Tim Cooijmans, Adam Roberts, Aaron Courville, Douglas Eck","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. In International conference on machine learning, pages 448-456. PMLR, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Jeong et al., 2019] Dasaem Jeong, Taegyun Kwon, Yoojin Kim, and Juhan Nam. Graph neural network for music score data and modeling expressive piano performance. In International Conference on Machine Learning, pages 3060-3070. PMLR, 2019.","title":"Graph neural network for music score data and modeling expressive piano performance","authors":"Jeong","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Karystinaios and Widmer, 2022] Emmanouil Karystinaios and Gerhard Widmer. Cadence detection in symbolic clas- sical music using graph neural networks, 2022.","title":"Cadence detection in symbolic classical music using graph neural networks","authors":"Widmer Karystinaios, Emmanouil Karystinaios, Gerhard Widmer","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Kingma and Welling, 2013] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Li et al., 2015] Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. Gated graph sequence neural networks. arXiv preprint arXiv:1511.05493, 2015.","title":"Gated graph sequence neural networks","authors":"Yujia Li, Daniel Tarlow, Marc Brockschmidt, Richard Zemel","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Li et al., 2018] Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In Thirty-Second AAAI confer- ence on artificial intelligence, 2018.","title":"Deeper insights into graph convolutional networks for semi-supervised learning","authors":"Li","year":"2018","venue":"Thirty-Second AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Li et al., 2019] Guohao Li, Matthias Muller, Ali Thabet, and Bernard Ghanem. Deepgcns: Can gcns go as deep as cnns? In Proceedings of the IEEE/CVF international conference on computer vision, pages 9267-9276, 2019.","title":"Deepgcns: Can gcns go as deep as cnns?","authors":"Li","year":"2019","venue":"Proceedings of the IEEE/CVF international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Liu et al., 2010] Xiao Fan Liu, K Tse Chi, and Michael Small. Complex network structure of musical composi- tions: Algorithmic generation of appealing music. Physica A: Statistical Mechanics and its Applications, 389(1):126- 132, 2010.","title":"Complex network structure of musical compositions: Algorithmic generation of appealing music","authors":"Fan Xiao, K Liu, Chi Tse, Michael Small","year":"2010","venue":"Physica A: Statistical Mechanics and its Applications","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Makhzani et al., 2015] Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Ad- versarial autoencoders. arXiv preprint arXiv:1511.05644, 2015.","title":"Adversarial autoencoders","authors":"Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, Brendan Frey","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Mogren, 2016] Olof Mogren. C-rnn-gan: Continuous re- current neural networks with adversarial training. arXiv preprint arXiv:1611.09904, 2016.","title":"C-rnn-gan: Continuous recurrent neural networks with adversarial training","authors":"Mogren","year":"2016","venue":"Olof Mogren","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Raffel, 2016] Colin Raffel. Learning-based methods for comparing sequences, with applications to audio-to-midi alignment and matching. Columbia University, 2016.","title":"Learning-based methods for comparing sequences, with applications to audio-to-midi alignment and matching","authors":"Colin Raffel","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Ramesh et al., 2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.","title":"Hierarchical text-conditional image generation with clip latents","authors":"Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, Mark Chen","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Roberts et al., 2018] Adam Roberts, Jesse Engel, Colin Raffel, Curtis Hawthorne, and Douglas Eck. A hierarchi- cal latent vector model for learning long-term structure in music. In International conference on machine learning, pages 4364-4373. PMLR, 2018.","title":"A hierarchical latent vector model for learning long-term structure in music","authors":"Adam Roberts, Jesse Engel, Colin Raffel, Curtis Hawthorne, Douglas Eck","year":"2018","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Schlichtkrull et al., 2018] Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convo- lutional networks. In European semantic web conference, pages 593-607. Springer, 2018.","title":"Modeling relational data with graph convolutional networks","authors":"Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den, Ivan Berg, Max Titov, Welling","year":"2018","venue":"European semantic web conference","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Simonetta et al., 2018] Federico Simonetta, Filippo Carnovalini, Nicola Orio, and Antonio Rodà. Symbolic music similarity through a graph-based representation. In Proceedings of the Audio Mostly 2018 on Sound in Immersion and Emotion, pages 1-7. 2018.","title":"Symbolic music similarity through a graph-based representation","authors":"Federico Simonetta, Filippo Carnovalini, Nicola Orio, Antonio Rodà","year":"2018","venue":"Proceedings of the Audio Mostly 2018 on Sound in Immersion and Emotion","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Simonovsky and Komodakis, 2017] Martin Simonovsky and Nikos Komodakis. Dynamic edge-conditioned filters in convolutional neural networks on graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3693-3702, 2017.","title":"Dynamic edge-conditioned filters in convolutional neural networks on graphs","authors":"Martin Simonovsky, Nikos Komodakis","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Valenti et al., 2020] Andrea Valenti, Antonio Carta, and Da- vide Bacciu. Learning style-aware symbolic music rep- resentations by adversarial autoencoders. arXiv preprint arXiv:2001.05494, 2020.","title":"Learning style-aware symbolic music representations by adversarial autoencoders","authors":"Andrea Valenti, Antonio Carta, Davide Bacciu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Valenti et al., 2021] Andrea Valenti, Stefano Berti, and Da- vide Bacciu. Calliope-a polyphonic music transformer. arXiv preprint arXiv:2107.05546, 2021.","title":"Calliope-a polyphonic music transformer","authors":"Andrea Valenti, Stefano Berti, Davide Bacciu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Wang et al., 2020] Ziyu Wang, Yiyi Zhang, Yixiao Zhang, Junyan Jiang, Ruihan Yang, Junbo Zhao, and Gus Xia. Pi- anotree vae: Structured representation learning for poly- phonic music. arXiv preprint arXiv:2008.07118, 2020.","title":"Pianotree vae: Structured representation learning for polyphonic music","authors":"Ziyu Wang, Yiyi Zhang, Yixiao Zhang, Junyan Jiang, Ruihan Yang, Junbo Zhao, Gus Xia","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Wu et al., 2020] Jian Wu, Xiaoguang Liu, Xiaolin Hu, and Jun Zhu. Popmnet: Generating structured pop music melodies using neural networks. Artificial Intelligence, 286:103303, 2020.","title":"Popmnet: Generating structured pop music melodies using neural networks","authors":"Jian Wu, Xiaoguang Liu, Xiaolin Hu, Jun Zhu","year":"2020","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Yang et al., 2017] Li-Chia Yang, Szu-Yu Chou, and Yi- Hsuan Yang. Midinet: A convolutional generative ad- versarial network for symbolic-domain music generation. arXiv preprint arXiv:1703.10847, 2017.","title":"Midinet: A convolutional generative adversarial network for symbolic-domain music generation","authors":"Li-Chia Yang, Szu-Yu Chou, Yi-Hsuan Yang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Zou et al., 2021] Yi Zou, Pei Zou, Yi Zhao, Kaixiang Zhang, Ran Zhang, and Xiaorui Wang. Melons: gen- erating melody with long-term structure using transform- ers and structure graph. arXiv preprint arXiv:2110.05020, 2021.","title":"Melons: generating melody with long-term structure using transformers and structure graph","authors":"Yi Zou, Pei Zou, Yi Zhao, Kaixiang Zhang, Ran Zhang, Xiaorui Wang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}