{"type":"ScholarlyArticle","title":"Musical Voice Separation as Link Prediction: Modeling a Musical Perception Task as a Multi-Trajectory Tracking Problem","authors":[{"name":"Emmanouil Karystinaios","slug":"emmanouil-karystinaios","given":"Emmanouil","family":"Karystinaios","affiliation":"Johannes Kepler University Linz , Austria"},{"name":"Francesco Foscarin","slug":"francesco-foscarin","given":"Francesco","family":"Foscarin","affiliation":"Johannes Kepler University Linz , Austria"},{"name":"Gerhard Widmer","slug":"gerhard-widmer","given":"Gerhard","family":"Widmer","affiliation":"Johannes Kepler University Linz , Austria; LIT AI Lab , Linz Institute of Technology , Austria"}],"abstract":"This paper targets the perceptual task of separating the different interacting voices, i.e., monophonic melodic streams, in a polyphonic musical piece. We target symbolic music, where notes are explicitly encoded, and model this task as a Multi-Trajectory Tracking (MTT) problem from discrete observations, i.e., notes in a pitch-time space. Our approach builds a graph from a musical piece, by creating one node for every note, and separates the melodic trajectories by predicting a link between two notes if they are consecutive in the same voice/stream. This kind of local, greedy prediction is made possible by node embeddings created by a heterogeneous graph neural network that can capture inter- and intra-trajectory information. Furthermore, we propose a new regularization loss that encourages the output to respect the MTT premise of at most one incoming and one outgoing link for every node, favoring monophonic (voice) trajectories; this loss function might also be useful in other general MTT scenarios. Our approach does not use domain-specific heuristics, is scalable to longer sequences and a higher number of voices, and can handle complex cases such as voice inversions and overlaps. We reach new state-of-the-art results for the voice separation task on classical music of different styles. All code, data, and pretrained models are available on https://github.com/manoskary/vocsep_ijcai2023","doi":"10.24963/ijcai.2023/430","url":"https://ijcai.org/proceedings/2023/430","pdf":"https://ijcai.org/proceedings/2023/0430.pdf","markdown":"https://ijcai.org/proceedings/2023/430.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/430","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":430,"pages":{"first":3866,"last":3874},"keywords":["Machine Learning: ML: Applications","Machine Learning: ML: Sequence and graph learning","Multidisciplinary Topics and Applications: MDA: Arts and creativity"],"videos":[],"references":[{"position":1,"raw":"[Aldwell et al., 2018] Edward Aldwell, Carl Schachter, and Allen Cadwallader. Harmony and voice leading. Cengage Learning, 2018.","title":"Harmony and voice leading","authors":"Aldwell","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Aljanaki et al., 2021] Anna Aljanaki, Stefano Kalonaris, Gi- anluca Micchi, and Eric Nichols. Mcma: A symbolic mul- titrack contrapuntal music archive. Empirical Musicology Review, 16(1):99-105, 2021.","title":"Mcma: A symbolic multitrack contrapuntal music archive","authors":"Anna Aljanaki, Stefano Kalonaris, Gianluca Micchi, Eric Nichols","year":"2021","venue":"Empirical Musicology Review","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brasó and Leal-Taixé, 2020] Guillem Brasó and Laura Leal- Taixé. Learning a neural solver for multiple object tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020.","title":"Guillem Brasó and Laura Leal-Taixé. Learning a neural solver for multiple object tracking","authors":"Leal-Taixé Brasó","year":"2020","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bresson and Laurent, 2018] Xavier Bresson and Thomas Laurent. An experimental study of neural networks for variable graphs. In Proceedings of the International Con- ference on Learning Representations, 2018.","title":"An experimental study of neural networks for variable graphs","authors":"Laurent ; Bresson, Xavier Bresson, Thomas Laurent","year":"2018","venue":"Proceedings of the International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Burkard and Cela, 1999] Rainer E Burkard and Eranda Cela. Linear assignment problems and extensions. In Handbook of combinatorial optimization, pages 75-149. Springer, 1999.","title":"Linear assignment problems and extensions","authors":"Rainer E Burkard, Eranda Cela","year":"1999","venue":"Handbook of combinatorial optimization","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cambouropoulos, 2006] Emilios Cambouropoulos. Voice separation: theoretical, perceptual and computational per- spectives. In Proceedings of the International Conference on Music Perception and Cognition (ICMPC). Citeseer, 2006.","title":"Emilios Cambouropoulos. Voice separation: theoretical, perceptual and computational perspectives","authors":"Cambouropoulos","year":"2006","venue":"Proceedings of the International Conference on Music Perception and Cognition (ICMPC)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cambouropoulos, 2008] Emilios Cambouropoulos. Voice and stream: Perceptual and computational modeling of voice separation. Music Perception, 26(1):75-94, 2008.","title":"Emilios Cambouropoulos. Voice and stream: Perceptual and computational modeling of voice separation","authors":"Cambouropoulos","year":"2008","venue":"Music Perception","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cancino-Chacón et al., 2022] Carlos Eduardo Cancino- Chacón, Silvan David Peter, Emmanouil Karystinaios, Francesco Foscarin, Maarten Grachten, and Gerhard Widmer. Partitura: A python package for symbolic music processing. In Proceedings of the Music Encoding Conference (MEC), Halifax, Canada, 2022.","title":"Partitura: A python package for symbolic music processing","authors":"Cancino-Chacón","year":"2022","venue":"Proceedings of the Music Encoding Conference (MEC)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Castnnón and Finn, 2011] Gregory Castnnón and Lucas Finn. Multi-target tracklet stitching through network flows. In Proceedings of the Aerospace Conference. IEEE, 2011.","title":"Gregory Castnnón and Lucas Finn. Multi-target tracklet stitching through network flows","authors":"Finn Castnnón","year":"2011","venue":"Proceedings of the Aerospace Conference","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Chew and Wu, 2004] Elaine Chew and Xiaodan Wu. Sepa- rating voices in polyphonic music: A contig mapping ap- proach. In Proceedings of the International Symposium on Computer Music Modeling and Retrieval. Springer, 2004.","title":"Separating voices in polyphonic music: A contig mapping approach","authors":"Wu ; ] Elaine Chew, Xiaodan Chew, Wu","year":"2004","venue":"Proceedings of the International Symposium on Computer Music Modeling and Retrieval","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Chong et al., 2009] Chee-Yee Chong, Greg Castanon, Nathan Cooprider, Shozo Mori, Ravi Ravichandran, and Robert Macior. Efficient multiple hypothesis tracking by track segment graph. In Proceedings of the International Conference on Information Fusion. IEEE, 2009.","title":"Efficient multiple hypothesis tracking by track segment graph","authors":"Chee-Yee Chong, Greg Castanon, Nathan Cooprider, Shozo Mori, Ravi Ravichandran, Robert Macior","year":"2009","venue":"Proceedings of the International Conference on Information Fusion","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Crouse, 2016] David F Crouse. On implementing 2d rect- angular assignment algorithms. IEEE Transactions on Aerospace and Electronic Systems, 52(4):1679-1696, 2016.","title":"On implementing 2d rectangular assignment algorithms","authors":"David F Crouse","year":"2016","venue":"IEEE Transactions on Aerospace and Electronic Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Duane and Pardo, 2009] Ben Duane and Bryan Pardo. Streaming from midi using constraint satisfaction optimiza- tion and sequence alignment. In Proceedings of the Inter- national Computer Music Conference (ICMC), 2009.","title":"Streaming from midi using constraint satisfaction optimization and sequence alignment","authors":"Duane, Pardo ; ] Ben, Duane, Bryan Pardo","year":"2009","venue":"Proceedings of the International Computer Music Conference (ICMC)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Dwivedi et al., 2020] Vijay Prakash Dwivedi, Chaitanya K Joshi, Thomas Laurent, Yoshua Bengio, and Xavier Bres- son. Benchmarking graph neural networks. arXiv preprint arXiv:2003.00982, 2020.","title":"Benchmarking graph neural networks","authors":"Vijay Prakash Dwivedi, K Chaitanya, Thomas Joshi, Yoshua Laurent, Xavier Bengio, Bresson","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Foscarin et al., 2022] Francesco Foscarin, Emmanouil Karystinaios, Silvan David Peter, Carlos Cancino-Chacón, Maarten Grachten, and Gerhard Widmer. The match file format: Encoding alignments between scores and performances. In Proceedings of the Music Encoding Conference (MEC), Halifax, Canada, 2022.","title":"The match file format: Encoding alignments between scores and performances","authors":"Francesco Foscarin, Emmanouil Karystinaios, Silvan David Peter, Carlos Cancino-Chacón, Maarten Grachten, Gerhard Widmer","year":"2022","venue":"Proceedings of the Music Encoding Conference (MEC)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Foscarin, 2020] Francesco Foscarin. The Musical Score: a challenging goal for automatic music transcription. PhD thesis, Paris, CNAM, 2020.","title":"The Musical Score: a challenging goal for automatic music transcription","authors":"Francesco Foscarin","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Gray and Bunescu, 2016] Patrick Gray and Razvan C Bunescu. A neural greedy model for voice separation in symbolic music. In Proceedings of the International Soci- ety for Music Information Retrieval Conference (ISMIR), 2016.","title":"A neural greedy model for voice separation in symbolic music","authors":"Patrick Gray, Razvan C Bunescu","year":"2016","venue":"Proceedings of the International Society for Music Information Retrieval Conference (ISMIR)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Hamilton et al., 2017] William L. Hamilton, Rex Ying, and Jure Leskovec. Representation learning on graphs: Meth- ods and applications. IEEE Data Engineering Bulletin, 40(3):52-74, 2017.","title":"Representation learning on graphs: Methods and applications","authors":"William L Hamilton, Rex Ying, Jure Leskovec","year":"2017","venue":"IEEE Data Engineering Bulletin","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Han et al., 2004] Mei Han, Wei Xu, Hai Tao, and Yihong Gong. An algorithm for multiple object trajectory tracking. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 1. IEEE, 2004.","title":"An algorithm for multiple object trajectory tracking","authors":"Mei Han, Wei Xu, Hai Tao, Yihong Gong","year":"2004","venue":"Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Hastie et al., 2009] Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer, 2009.","title":"The elements of statistical learning: data mining, inference, and prediction","authors":"Trevor Hastie, Robert Tibshirani, Jerome H Friedman, Jerome H Friedman","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Hsiao and Su, 2021] Yo-Wei Hsiao and Li Su. Learning note- to-note affinity for voice segregation and melody line iden- tification of symbolic music data. In Proceedings of the International Society for Music Information Retrieval Con- ference (ISMIR), 2021.","title":"Learning noteto-note affinity for voice segregation and melody line identification of symbolic music data","authors":"Su ; ] Yo-Wei Hsiao, Li Hsiao, Su","year":"2021","venue":"Proceedings of the International Society for Music Information Retrieval Conference (ISMIR)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Huron, 2001] David Huron. Tone and voice: A derivation of the rules of voice-leading from perceptual principles. Music Perception, 19(1):1-64, 2001.","title":"Tone and voice: A derivation of the rules of voice-leading from perceptual principles","authors":"David Huron","year":"2001","venue":"Music Perception","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Jeong et al., 2019] Dasaem Jeong, Taegyun Kwon, Yoojin Kim, and Juhan Nam. Graph neural network for music score data and modeling expressive piano performance. In Proceedings of the International Conference on Machine Learning (ICML), 2019.","title":"Graph neural network for music score data and modeling expressive piano performance","authors":"Jeong","year":"2019","venue":"Proceedings of the International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Jordanous, 2008] Anna Jordanous. Voice separation in poly- phonic music: A data-driven approach. In Proceedings of the International Computer Music Conference (ICMC), 2008.","title":"Voice separation in polyphonic music: A data-driven approach","authors":"Anna Jordanous","year":"2008","venue":"Proceedings of the International Computer Music Conference (ICMC)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Karystinaios and Widmer, 2022] Emmanouil Karystinaios and Gerhard Widmer. Cadence detection in symbolic clas- sical music using graph neural networks. In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), 2022.","title":"Cadence detection in symbolic classical music using graph neural networks","authors":"Widmer Karystinaios, Emmanouil Karystinaios, Gerhard Widmer","year":"2022","venue":"Proceedings of the International Society for Music Information Retrieval Conference (ISMIR)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Kilian and Hoos, 2002] Jürgen Kilian and Holger H Hoos. Voice separation-a local optimization approach. In Pro- ceedings of the International Society for Music Information Retrieval Conference (ISMIR). Citeseer, 2002.","title":"Voice separation-a local optimization approach","authors":"Hoos Kilian, Jürgen Kilian, H Holger, Hoos","year":"2002","venue":"Proceedings of the International Society for Music Information Retrieval Conference (ISMIR)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Liu et al., 2022] He Liu, Tao Wang, Congyan Lang, Songhe Feng, Yi Jin, and Yidong Li. Glan: A graph-based linear assignment network. arXiv preprint arXiv:2201.02057, 2022.","title":"Glan: A graph-based linear assignment network","authors":"Liu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Ma et al., 2022] Xichu Ma, Xiao Liu, Bowen Zhang, and Ye Wang. Robust melody track identification in symbolic music. In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), 2022.","title":"Robust melody track identification in symbolic music","authors":"Xichu Ma, Xiao Liu, Bowen Zhang, Ye Wang","year":"2022","venue":"Proceedings of the International Society for Music Information Retrieval Conference (ISMIR)","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Madsen and Widmer, 2006] Søren Tjagvad Madsen and Ger- hard Widmer. Separating voices in midi. In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR). Citeseer, 2006.","title":"Separating voices in midi","authors":"Widmer Madsen, Gerhard Widmer","year":"2006","venue":"Proceedings of the International Society for Music Information Retrieval Conference (ISMIR)","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[McLeod and Steedman, 2016] Andrew McLeod and Mark Steedman. Hmm-based voice separation of midi perfor- mance. Journal of New Music Research, 45(1):17-26, 2016.","title":"Hmm-based voice separation of midi performance","authors":"Andrew Mcleod, Mark Steedman","year":"2016","venue":"Journal of New Music Research","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Rafailidis et al., 2009] Dimitris Rafailidis, Emilios Cam- bouropoulos, and Yannis Manolopoulos. Musical voice integration/segregation: Visa revisited. In Proceedings of the Sound and Music Computing Conference (SMC), 2009.","title":"Musical voice integration/segregation: Visa revisited","authors":"Dimitris Rafailidis, Emilios Cambouropoulos, Yannis Manolopoulos","year":"2009","venue":"Proceedings of the Sound and Music Computing Conference (SMC)","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Schlichtkrull et al., 2018] Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolu- tional networks. In Proceedings of the European Semantic Web Conference. Springer, 2018.","title":"Modeling relational data with graph convolutional networks","authors":"Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den, Ivan Berg, Max Titov, Welling","year":"2018","venue":"Proceedings of the European Semantic Web Conference","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Shooner et al., 2010] Christopher Shooner, Srimant P Tripa- thy, Harold E Bedell, and Haluk Ögmen. High-capacity, transient retention of direction-of-motion information for multiple moving objects. Journal of Vision, 10(6):1-20, 2010.","title":"High-capacity, transient retention of direction-of-motion information for multiple moving objects","authors":"Christopher Shooner, P Srimant, Harold E Tripathy, Haluk Bedell, Ögmen","year":"2010","venue":"Journal of Vision","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Temperley, 2008] David Temperley. A probabilistic model of melody perception. Cognitive Science, 32(2):418-444, 2008.","title":"A probabilistic model of melody perception","authors":"David Temperley","year":"2008","venue":"Cognitive Science","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Van der Merwe and De Villiers, 2016] Lodewyk Van der Merwe and Pieter De Villiers. Comparative investigation into viterbi based and multiple hypothesis based track stitch- ing. IET Radar, Sonar & Navigation, 10(9):1575-1582, 2016.","title":"Comparative investigation into viterbi based and multiple hypothesis based track stitching","authors":"Merwe Van Der, De Villiers, ; Lodewyk Van Der Merwe, Pieter De Villiers","year":"2016","venue":"IET Radar, Sonar & Navigation","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Wang et al., 2021] Yongxin Wang, Kris Kitani, and Xinshuo Weng. Joint object detection and multi-object tracking with graph neural networks. In Proceedings of the IEEE Inter- national Conference on Robotics and Automation (ICRA). IEEE, 2021.","title":"Joint object detection and multi-object tracking with graph neural networks","authors":"Wang","year":"2021","venue":"Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Weng et al., 2021] Xinshuo Weng, Ye Yuan, and Kris Kitani. Ptp: Parallelized tracking and prediction with graph neu- ral networks and diversity sampling. IEEE Robotics and Automation Letters, 6(3):4640-4647, 2021.","title":"Ptp: Parallelized tracking and prediction with graph neu-ral networks and diversity sampling","authors":"Weng","year":"2021","venue":"IEEE Robotics and Automation Letters","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Xu et al., 2018] Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In Proceedings of the International Conference on Machine Learning (ICML), 2018.","title":"Representation learning on graphs with jumping knowledge networks","authors":"Xu","year":"2018","venue":"Proceedings of the International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}