{"type":"ScholarlyArticle","title":"Discrete Diffusion Probabilistic Models for Symbolic Music Generation","authors":[{"name":"Matthias Plasser","slug":"matthias-plasser","given":"Matthias","family":"Plasser","affiliation":"Institute of Computational Perception , Johannes Kepler University Linz , Austria"},{"name":"Silvan Peter","slug":"silvan-peter","given":"Silvan","family":"Peter","affiliation":"Institute of Computational Perception , Johannes Kepler University Linz , Austria"},{"name":"Gerhard Widmer","slug":"gerhard-widmer","given":"Gerhard","family":"Widmer","affiliation":"Institute of Computational Perception , Johannes Kepler University Linz , Austria; LIT AI Lab , Linz Institute of Technology , Austria"}],"abstract":"Denoising Diffusion Probabilistic Models (DDPMs) have made great strides in generating high-quality samples in both discrete and continuous domains.\r\nHowever, Discrete DDPMs (D3PMs) have yet to be applied to the domain of Symbolic Music.\r\nThis work presents the direct generation of Polyphonic Symbolic Music using D3PMs.\r\nOur model exhibits state-of-the-art sample quality, according to current quantitative evaluation metrics, and allows for flexible infilling at the note level.\r\nWe further show, that our models are accessible to post-hoc classifier guidance, widening the scope of possible applications.\r\nHowever, we also cast a critical view on quantitative evaluation of music sample quality via statistical metrics, and present a simple algorithm that can confound our metrics with completely spurious, non-musical samples.","doi":"10.24963/ijcai.2023/648","url":"https://ijcai.org/proceedings/2023/648","pdf":"https://ijcai.org/proceedings/2023/0648.pdf","markdown":"https://ijcai.org/proceedings/2023/648.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/648","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"AI and Arts","track":"AI and Arts","number":648,"pages":{"first":5842,"last":5850},"keywords":["Theory and philosophy of arts and creativity in AI systems: Autonomous creative or artistic AI","Methods and resources: Machine learning, deep learning, neural models, reinforcement learning","Application domains: Music and sound","Theory and philosophy of arts and creativity in AI systems: Evaluation of artistic or creative outputs produced by AI Systems"],"videos":[],"references":[{"position":1,"raw":"[Agres et al., 2016] Kat Agres, Jamie Forth, and Geraint Wiggins. Evaluation of musical creativity and musical metacreation systems. Comput. Entertain., 14(3):3:1- 3:33, December 2016.","title":"Evaluation of musical creativity and musical metacreation systems","authors":"Agres","year":"2016","venue":"Comput. Entertain","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Aiello, 2016] Luigia Carlucci Aiello. The multifaceted im- pact of ada lovelace in the digital age. Artificial Intelli- gence, 235:58-62, 2016.","title":"The multifaceted impact of ada lovelace in the digital age","authors":"Luigia Carlucci, Aiello","year":"2016","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Austin et al., 2021] Jacob Austin, Daniel Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Struc- tured denoising diffusion models in discrete state-spaces. In Advances in Neural Information Processing Systems, volume 34, pages 17981-17993.","title":"Structured denoising diffusion models in discrete state-spaces","authors":"Jacob Austin, Daniel Johnson, Jonathan Ho, Daniel Tarlow, Rianne Van Den, Berg","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"Curran Associates, Inc., 2021.","title":null,"authors":"Curran Associates, Inc","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bond-Taylor et al., 2022] Sam Bond-Taylor, Peter Hessey, Hiroshi Sasaki, Toby Breckon, and Chris Willcocks. Un- leashing transformers: Parallel token prediction with dis- crete absorbing diffusion for fast high-resolution image generation from vector-quantized codes. In Shai Avi- dan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision - ECCV 2022, pages 170-188, Cham, 2022. Springer Na- ture Switzerland.","title":"Unleashing transformers: Parallel token prediction with discrete absorbing diffusion for fast high-resolution image generation from vector-quantized codes","authors":"Sam Bond-Taylor, Peter Hessey, Hiroshi Sasaki, Toby Breckon, Chris Willcocks","year":"2022","venue":"Computer Vision -ECCV 2022","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Brown et al., 2020] Tom Brown, Benjamin Mann, Nick Ry- der, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learn- ers. In Advances in Neural Information Processing Sys- tems, volume 33, pages 1877-1901.","title":"Language models are few-shot learners","authors":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam Mccandlish, Alec Radford, Ilya Sutskever, Dario Amodei","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"Curran Associates, Inc., 2020.","title":null,"authors":"Curran Associates, Inc","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chou et al., 2021] Yi-Hui Chou, I-Chun Chen, Chin-Jui Chang, Joann Ching, and Yi-Hsuan Yang. Midibert-piano: Large-scale pre-training for symbolic music understand- ing. arXiv preprint arXiv:2107.05223, 2021.","title":"Midibert-piano: Large-scale pre-training for symbolic music understanding","authors":"Yi-Hui Chou, I-Chun Chen, Chin-Jui Chang, Joann Ching, Yi-Hsuan Yang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Civit et al., 2022] Miguel Civit, Javier Civit-Masot, Fran- cisco Cuadrado, and Maria Escalona. A systematic review of artificial intelligence-based music generation: Scope, applications, and future trends. Expert Systems with Ap- plications, page 118190, 2022.","title":"A systematic review of artificial intelligence-based music generation: Scope, applications, and future trends","authors":"Civit","year":"2022","venue":"Expert Systems with Applications","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understand- ing. pages 4171-4186, June 2019.","title":"BERT: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Dhariwal and Nichol, 2021] Prafulla Dhariwal and Alexan- der Nichol. Diffusion models beat gans on image syn- thesis. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neu- ral Information Processing Systems, volume 34, pages 8780-8794.","title":"Diffusion models beat gans on image synthesis","authors":"Nichol ; Dhariwal, Alexander Dhariwal, Nichol","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"Curran Associates, Inc., 2021.","title":null,"authors":"Curran Associates, Inc","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial networks. Advances in Neural Informa- tion Processing Systems, 2014.","title":"Generative adversarial networks","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hawthorne et al., 2022] Curtis Hawthorne, Ian Simon, Adam Roberts, Neil Zeghidour, Josh Gardner, Ethan Manilow, and Jesse Engel. Multi-instrument music syn- thesis with spectrogram diffusion. International Society for Music Information Retrieval, 2022.","title":"Multi-instrument music synthesis with spectrogram diffusion","authors":"Curtis Hawthorne, Ian Simon, Adam Roberts, Neil Zeghidour, Josh Gardner, Ethan Manilow, Jesse Engel","year":"2022","venue":"International Society for Music Information Retrieval","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ho et al., 2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pages 6840-6851.","title":"Denoising diffusion probabilistic models","authors":"Jonathan Ho, Ajay Jain, Pieter Abbeel","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"Curran Associates, Inc., 2020.","title":null,"authors":"Curran Associates, Inc","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Hoogeboom et al., 2021] Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 12454-12465.","title":"Argmax flows and multinomial diffusion: Learning categorical distributions","authors":"Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, Max Welling","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"Curran Associates, Inc., 2021.","title":null,"authors":"Curran Associates, Inc","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Hsiao et al., 2021] Wen-Yi Hsiao, Jen-Yu Liu, Yin-Cheng Yeh, and Yi-Hsuan Yang. Compound word transformer: Learning to compose full-song music over dynamic di- rected hypergraphs. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 35, pages 178-186, 2021.","title":"Compound word transformer: Learning to compose full-song music over dynamic directed hypergraphs","authors":"Wen-Yi Hsiao, Jen-Yu Liu, Yin-Cheng Yeh, Yi-Hsuan Yang","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Kingma and Welling, 2014] Diederik Kingma and Max Welling. Auto-Encoding Variational Bayes. In 2nd Inter- national Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.","title":"Auto-Encoding Variational Bayes","authors":"Diederik Kingma, Max Welling","year":"2014","venue":"2nd International Conference on Learning Representations, ICLR 2014","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Kong et al., 2021] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versa- tile diffusion model for audio synthesis. In International Conference on Learning Representations, 2021.","title":"Diffwave: A versatile diffusion model for audio synthesis","authors":"Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, Bryan Catanzaro","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Matejka and Fitzmaurice, 2017] Justin Matejka and George Fitzmaurice. Same stats, different graphs: generating datasets with varied appearance and identical statistics through simulated annealing. In Proceedings of the 2017 CHI conference on human factors in computing systems, pages 1290-1294, 2017.","title":"Same stats, different graphs: generating datasets with varied appearance and identical statistics through simulated annealing","authors":"Fitzmaurice Matejka, Justin Matejka, George Fitzmaurice","year":"2017","venue":"Proceedings of the 2017 CHI conference on human factors in computing systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Mittal et al., 2021] Gautam Mittal, Jesse Engel, Curtis Hawthorne, and Ian Simon. Symbolic music generation with diffusion models. In International Society for Music Information Retrieval, 2021.","title":"Symbolic music generation with diffusion models","authors":"Gautam Mittal, Jesse Engel, Curtis Hawthorne, Ian Simon","year":"2021","venue":"International Society for Music Information Retrieval","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Naeem et al., 2020] Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reli- able fidelity and diversity metrics for generative models. In International Conference on Machine Learning, pages 7176-7185. PMLR, 2020.","title":"Reliable fidelity and diversity metrics for generative models","authors":"Naeem","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Pearce and Wiggins, 2001] Marcus Pearce and Geraint Wiggins. Towards a framework for the evaluation of ma- chine compositions. In Proc. Symp. AI and Creativity in the Arts and Sciences, 2001.","title":"Towards a framework for the evaluation of machine compositions","authors":"Pearce, Wiggins, Marcus Pearce, Geraint Wiggins","year":"2001","venue":"Proc. Symp. AI and Creativity in the Arts and Sciences","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Raffel, 2016] Colin Raffel. Learning-Based Methods for Comparing Sequences, with Applications to Audio-to- MIDI Alignment and Matching. PhD thesis, Columbia University, 2016.","title":"Learning-Based Methods for Comparing Sequences, with Applications to Audio-to-MIDI Alignment and Matching","authors":"Colin Raffel","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Roberts et al., 2018] Adam Roberts, Jesse Engel, Colin Raffel, Curtis Hawthorne, and Douglas Eck. A hierarchi- cal latent vector model for learning long-term structure in music. In Jennifer Dy and Andreas Krause, editors, Pro- ceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4364-4373. PMLR, 10-15 Jul 2018.","title":"A hierarchical latent vector model for learning long-term structure in music","authors":"Adam Roberts, Jesse Engel, Colin Raffel, Curtis Hawthorne, Douglas Eck","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Sohl-Dickstein et al., 2015] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermody- namics. In Francis Bach and David Blei, editors, Pro- ceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learn- ing Research, pages 2256-2265, Lille, France, 07-09 Jul 2015. PMLR.","title":"Deep unsupervised learning using nonequilibrium thermodynamics","authors":"Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, Surya Ganguli","year":"2015","venue":"Proceedings of the 32nd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Sturm and Ben-Tal, 2017] Bob Sturm and Oded Ben-Tal. Taking the models back to music practice: Evaluating gen- erative transcription models built using deep learning. J. Creative Music Systems, 2(1), 2017.","title":"Taking the models back to music practice: Evaluating generative transcription models built using deep learning","authors":"Ben-Tal Sturm, ;, Bob Sturm, Oded Ben-Tal","year":"2017","venue":"J. Creative Music Systems","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Varela et al., 2014] Diego Varela, Giacomo Benedetto, and Jose Manuel Sanchez-Santos. Editorial statement: Lessons from goodhart's law for the management of the journal. European Journal of Government and Economics, 3(2):100-103, 2014.","title":"Editorial statement: Lessons from goodhart's law for the management of the journal","authors":"Varela","year":"2014","venue":"European Journal of Government and Economics","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yang et al., 2017] Li-Chia Yang, Szu-Yu Chou, and Yi- Hsuan Yang. Midinet: A convolutional generative adver- sarial network for symbolic-domain music generation us- ing 1d and 2d conditions. ISMIR 2017, pages 324-331, 2017.","title":"Midinet: A convolutional generative adversarial network for symbolic-domain music generation using 1d and 2d conditions","authors":"Li-Chia Yang, Szu-Yu Chou, Yi-Hsuan Yang","year":"2017","venue":"ISMIR 2017","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zeng et al., 2021] Mingliang Zeng, Xu Tan, Rui Wang, Ze- qian Ju, Tao Qin, and Tie-Yan Liu. Musicbert: Symbolic music understanding with large-scale pre-training. arXiv preprint arXiv:2106.05630, 2021.","title":"Musicbert: Symbolic music understanding with large-scale pre-training","authors":"Mingliang Zeng, Xu Tan, Rui Wang, Zeqian Ju, Tao Qin, Tie-Yan Liu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}