{"type":"ScholarlyArticle","title":"End-to-End Real-World Polyphonic Piano Audio-to-Score Transcription with Hierarchical Decoding","authors":[{"name":"Wei Zeng","slug":"wei-zeng","given":"Wei","family":"Zeng","affiliation":"Integrative Sciences and Engineering Programme , NUS Graduate School"},{"name":"Xian He","slug":"xian-he","given":"Xian","family":"He","affiliation":"School of Computing , National University of Singapore"},{"name":"Ye Wang","slug":"ye-wang","given":"Ye","family":"Wang","affiliation":"Integrative Sciences and Engineering Programme , NUS Graduate School; School of Computing , National University of Singapore"}],"abstract":"Piano audio-to-score transcription (A2S) is an important yet underexplored task with extensive applications for music composition, practice, and analysis. However, existing end-to-end piano A2S systems faced difficulties in retrieving bar-level information such as key and time signatures, and have been trained and evaluated with only synthetic data. To address these limitations, we propose a sequence-to-sequence (Seq2Seq) model with a hierarchical decoder that aligns with the hierarchical structure of musical scores, enabling the transcription of score information at both the bar and note levels by multi-task learning. To bridge the gap between synthetic data and recordings of human performance, we propose a two-stage training scheme, which involves pre-training the model using an expressive performance rendering (EPR) system on synthetic audio, followed by fine-tuning the model using recordings of human performance. To preserve the voicing structure for score reconstruction, we propose a pre-processing method for **Kern scores in scenarios with an unconstrained number of voices. Experimental results support the effectiveness of our proposed approaches, in terms of both transcription performance on synthetic audio data in comparison to the current state-of-the-art, and the first experiment on human recordings.","doi":"10.24963/ijcai.2024/862","url":"https://ijcai.org/proceedings/2024/862","pdf":"https://ijcai.org/proceedings/2024/0862.pdf","markdown":"https://ijcai.org/proceedings/2024/862.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/862","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"AI, Arts & Creativity","track":"AI, Arts & Creativity","number":862,"pages":{"first":7788,"last":7795},"keywords":["Methods and resources: Machine learning, deep learning, neural models, reinforcement learning","Application domains: Music and sound"],"videos":[],"references":[{"position":1,"raw":"[Arroyo et al., 2022] Víctor Arroyo, Jose J Valero-Mas, Jorge Calvo-Zaragoza, and Antonio Pertusa. Neural audio- to-score music transcription for unconstrained polyphony using compact output representations. In ICASSP 2022- 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4603-4607. IEEE, 2022.","title":"Neural audioto-score music transcription for unconstrained polyphony using compact output representations","authors":"Víctor Arroyo, Jose J Valero-Mas, Jorge Calvo-Zaragoza, Antonio Pertusa","year":"2022","venue":"ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Carvalho and Smaragdis, 2017] Ralf Gunter Correa Car- valho and Paris Smaragdis. Towards end-to-end poly- phonic music transcription: Transforming music audio di- rectly to a score. In 2017 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pages 151-155. IEEE, 2017.","title":"Towards end-to-end polyphonic music transcription: Transforming music audio directly to a score","authors":"Smaragdis ; Carvalho, Paris Valho, Smaragdis","year":"2017","venue":"IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cogliati et al., 2016] Andrea Cogliati, David Temperley, and Zhiyao Duan. Transcribing human piano perfor- mances into music notation. In ISMIR, pages 758-764, 2016.","title":"Transcribing human piano performances into music notation","authors":"Andrea Cogliati, David Temperley, Zhiyao Duan","year":"2016","venue":"ISMIR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Foscarin et al., 2020] Francesco Foscarin, Andrew Mcleod, Philippe Rigaux, Florent Jacquemard, and Masahiko Sakai. Asap: a dataset of aligned scores and performances for piano transcription. In International Society for Mu- sic Information Retrieval Conference, pages 534-541. IS- MIR, November 2020.","title":"Asap: a dataset of aligned scores and performances for piano transcription","authors":"Francesco Foscarin, Andrew Mcleod, Philippe Rigaux, Florent Jacquemard, Masahiko Sakai","year":"2020","venue":"International Society for Music Information Retrieval Conference","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hawthorne et al., 2019] Curtis Hawthorne, Andriy Stasyuk, Adam Roberts, Ian Simon, Cheng-Zhi Anna Huang, Sander Dieleman, Erich Elsen, Jesse Engel, and Douglas Eck. Enabling factorized piano music modeling and gen- eration with the MAESTRO dataset. In International Con- ference on Learning Representations, 2019.","title":"Enabling factorized piano music modeling and generation with the MAESTRO dataset","authors":"Curtis Hawthorne, Andriy Stasyuk, Adam Roberts, Ian Simon, Cheng-Zhi Anna Huang, Sander Dieleman, Erich Elsen, Jesse Engel, Douglas Eck","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Jeong et al., 2019a] Dasaem Jeong, Taegyun Kwon, Yoojin Kim, Kyogu Lee, and Juhan Nam. Virtuosonet: A hier- archical rnn-based system for modeling expressive piano performance. In ISMIR, pages 908-915, 2019.","title":"Virtuosonet: A hierarchical rnn-based system for modeling expressive piano performance","authors":"Jeong","year":"2019","venue":"ISMIR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jeong et al., 2019b] Dasaem Jeong, Taegyun Kwon, Yoojin Kim, and Juhan Nam. Graph neural network for music score data and modeling expressive piano performance. In International conference on machine learning, pages 3060-3070. PMLR, 2019.","title":"Graph neural network for music score data and modeling expressive piano performance","authors":"Jeong","year":"2019","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Koelsch et al., 2013] Stefan Koelsch, Martin Rohrmeier, Renzo Torrecuso, and Sebastian Jentschke. Processing of hierarchical syntactic structure in music. Proceedings of the National Academy of Sciences, 110(38):15443-15448, 2013.","title":"Processing of hierarchical syntactic structure in music","authors":"Stefan Koelsch, Martin Rohrmeier, Renzo Torrecuso, Sebastian Jentschke","year":"2013","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Krumhansl et al., 1982] Carol L Krumhansl, Jamshed Bharucha, and Mary A Castellano. Key distance effects on perceived harmonic structure in music. Perception & Psychophysics, 32(2):96-108, 1982.","title":"Key distance effects on perceived harmonic structure in music","authors":"Jamshed Carol L Krumhansl, Mary A Bharucha, Castellano","year":"1982","venue":"Perception & Psychophysics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lamb et al., 2016] Alex M Lamb, Anirudh Goyal Alias Parth Goyal, Ying Zhang, Saizheng Zhang, Aaron C Courville, and Yoshua Bengio. Professor forcing: A new algorithm for training recurrent networks. Advances in neural information processing systems, 29, 2016.","title":"Professor forcing: A new algorithm for training recurrent networks","authors":"Alex M Lamb, Anirudh Goyal Alias, Parth Goyal, Ying Zhang, Saizheng Zhang, Aaron C Courville, Yoshua Bengio","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2021] Lele Liu, Veronica Morfi, and Emmanouil Benetos. Joint multi-pitch detection and score transcrip- tion for polyphonic piano music. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 281-285. IEEE, 2021.","title":"Joint multi-pitch detection and score transcription for polyphonic piano music","authors":"Lele Liu, Veronica Morfi, Emmanouil Benetos","year":"2021","venue":"ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[McLeod and Steedman, 2018] Andrew McLeod and Mark Steedman. Evaluating automatic polyphonic music tran- scription. In ISMIR, pages 42-49, 2018.","title":"Evaluating automatic polyphonic music transcription","authors":"Andrew Mcleod, Mark Steedman","year":"2018","venue":"ISMIR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[McLeod, 2019] Andrew McLeod. Evaluating non-aligned musical score transcriptions with mv2h. arXiv preprint arXiv:1906.00566, 2019.","title":"Evaluating non-aligned musical score transcriptions with mv2h","authors":"Andrew Mcleod","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Nakamura et al., 2018] Eita Nakamura, Emmanouil Bene- tos, Kazuyoshi Yoshii, and Simon Dixon. Towards com- plete polyphonic music transcription: Integrating multi- pitch detection and rhythm quantization. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 101-105. IEEE, 2018.","title":"Towards complete polyphonic music transcription: Integrating multipitch detection and rhythm quantization","authors":"Eita Nakamura, Emmanouil Benetos, Kazuyoshi Yoshii, Simon Dixon","year":"2018","venue":"IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ravanelli et al., 2021] Mirco Ravanelli, Titouan Parcollet, Peter Plantinga, Aku Rouhe, Samuele Cornell, Loren Lu- gosch, Cem Subakan, Nauman Dawalatabad, Abdelwahab Heba, Jianyuan Zhong, et al. Speechbrain: A general- purpose speech toolkit. arXiv preprint arXiv:2106.04624, 2021.","title":"Speechbrain: A generalpurpose speech toolkit","authors":"Mirco Ravanelli, Titouan Parcollet, Peter Plantinga, Aku Rouhe, Samuele Cornell, Loren Lugosch, Cem Subakan, Nauman Dawalatabad, Abdelwahab Heba, Jianyuan Zhong","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Renault et al., 2023] Lenny Renault, Rémi Mignot, and Axel Roebel. Expressive piano performance rendering from unpaired data. In International Conference on Digi- tal Audio Effects (DAFx23), 2023.","title":"Expressive piano performance rendering from unpaired data","authors":"Renault","year":"2023","venue":"International Conference on Digital Audio Effects (DAFx23)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Román et al., 2018] Miguel A Román, Antonio Pertusa, and Jorge Calvo-Zaragoza. An end-to-end framework for audio-to-score music transcription on monophonic ex- cerpts. In ISMIR, pages 34-41, 2018.","title":"An end-to-end framework for audio-to-score music transcription on monophonic excerpts","authors":"Antonio Miguel A Román, Jorge Pertusa, Calvo-Zaragoza","year":"2018","venue":"ISMIR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Román et al., 2019] Miguel A Román, Antonio Pertusa, and Jorge Calvo-Zaragoza. A holistic approach to poly- phonic music transcription with neural networks. arXiv preprint arXiv:1910.12086, 2019.","title":"A holistic approach to polyphonic music transcription with neural networks","authors":"Antonio Miguel A Román, Jorge Pertusa, Calvo-Zaragoza","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Schörkhuber et al., 2014] Christian Schörkhuber, Anssi Klapuri, Nicki Holighaus, and Monika Dörfler. A matlab toolbox for efficient perfect reconstruction time-frequency transforms with log-frequency resolution. In Audio Engineering Society Conference: 53rd International Conference: Semantic Audio. Audio Engineering Society, 2014.","title":"Christian Schörkhuber, Anssi Klapuri, Nicki Holighaus, and Monika Dörfler. A matlab toolbox for efficient perfect reconstruction time-frequency transforms with log-frequency resolution","authors":"Schörkhuber","year":"2014","venue":"Audio Engineering Society Conference: 53rd International Conference: Semantic Audio","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shibata et al., 2021] Kentaro Shibata, Eita Nakamura, and Kazuyoshi Yoshii. Non-local musical statistics as guides for audio-to-score piano transcription. Information Sci- ences, 566:262-280, 2021.","title":"Non-local musical statistics as guides for audio-to-score piano transcription","authors":"Kentaro Shibata, Eita Nakamura, Kazuyoshi Yoshii","year":"2021","venue":"Information Sciences","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhang and Yang, 2021] Yu Zhang and Qiang Yang. A sur- vey on multi-task learning. IEEE Transactions on Knowl- edge and Data Engineering, 34(12):5586-5609, 2021.","title":"A survey on multi-task learning","authors":"Yu Zhang, Qiang Yang","year":"2021","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}