{"type":"ScholarlyArticle","title":"JEPOO: Highly Accurate Joint Estimation of Pitch, Onset and Offset for Music Information Retrieval","authors":[{"name":"Haojie Wei","slug":"haojie-wei","given":"Haojie","family":"Wei","affiliation":"School of Information , Renmin University of China , Beijing , China"},{"name":"Jun Yuan","slug":"jun-yuan","given":"Jun","family":"Yuan","affiliation":"Huawei Noah's Ark Lab , Shenzhen , China"},{"name":"Rui Zhang","slug":"rui-zhang","given":"Rui","family":"Zhang","affiliation":"Tsinghua University ‡"},{"name":"Yueguo Chen","slug":"yueguo-chen","given":"Yueguo","family":"Chen","affiliation":"School of Information , Renmin University of China , Beijing , China"},{"name":"Gang Wang","slug":"gang-wang","given":"Gang","family":"Wang","affiliation":"Huawei Noah's Ark Lab , Shenzhen , China"}],"abstract":"Melody extraction is a core task in music information retrieval, and the estimation of pitch, onset and offset are key sub-tasks in melody extraction. Existing methods have limited accuracy, and work for only one type of data, either single-pitch or multi-pitch. In this paper, we propose a highly accurate method for joint estimation of pitch, onset and offset, named JEPOO. We address the challenges of joint learning optimization and handling both single-pitch and multi-pitch data through novel model design and a new optimization technique named Pareto modulated loss with loss weight regularization. This is the first method that can accurately handle both single-pitch and multi-pitch music data, and even a mix of them. A comprehensive experimental study on a wide range of real datasets shows that JEPOO outperforms state-of-the-art methods by up to 10.6\\%, 8.3\\% and 10.3\\% for the prediction of Pitch, Onset and Offset, respectively, and JEPOO is robust for various types of data and instruments. The ablation study validates the effectiveness of each component of JEPOO.","doi":"10.24963/ijcai.2023/544","url":"https://ijcai.org/proceedings/2023/544","pdf":"https://ijcai.org/proceedings/2023/0544.pdf","markdown":"https://ijcai.org/proceedings/2023/544.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/544","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Multidisciplinary Topics and Applications","track":"Main Track","number":544,"pages":{"first":4892,"last":4902},"keywords":["Multidisciplinary Topics and Applications: MDA: Arts and creativity","Multidisciplinary Topics and Applications: MDA: Other","Multidisciplinary Topics and Applications: MDA: Entertainment"],"videos":[],"references":[{"position":1,"raw":"[Benetos et al., 2018] Emmanouil Benetos, Simon Dixon, Zhiyao Duan, and Sebastian Ewert. Automatic music tran- scription: An overview. IEEE Signal Processing Maga- zine, pages 20-30, 2018.","title":"Automatic music transcription: An overview","authors":"Emmanouil Benetos, Simon Dixon, Zhiyao Duan, Sebastian Ewert","year":"2018","venue":"IEEE Signal Processing Magazine","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bittner et al., 2018] Rachel M Bittner, Brian McFee, and Juan P Bello. Multitask learning for fundamen- tal frequency estimation in music. arXiv preprint arXiv:1809.00381, 2018.","title":"Multitask learning for fundamental frequency estimation in music","authors":"Rachel M Bittner, Brian Mcfee, Juan P Bello","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2018] Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In Proceedings of the International conference on machine learning (ICML), pages 794-803, 2018.","title":"Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks","authors":"Vijay Zhao Chen, Chen-Yu Badrinarayanan, Andrew Lee, Rabinovich","year":"2018","venue":"Proceedings of the International conference on machine learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Choi et al., 2017] Keunwoo Choi, György Fazekas, Mark Sandler, and Kyunghyun Cho. Transfer learning for mu- sic classification and regression tasks. arXiv preprint arXiv:1703.09179, 2017.","title":"Transfer learning for music classification and regression tasks","authors":"Choi","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[De Cheveigné and Kawahara, 2002] Alain De Cheveigné and Hideki Kawahara. Yin, a fundamental frequency esti- mator for speech and music. The Journal of the Acoustical Society of America (JASA), pages 1917-1930, 2002.","title":"Yin, a fundamental frequency estimator for speech and music","authors":"De Cheveigné, Alain De Kawahara, Hideki Cheveigné, Kawahara","year":"2002","venue":"The Journal of the Acoustical Society of America","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dubnowski et al., 1976] John Dubnowski, Ronald Schafer, and Lawrence Rabiner. Real-time digital hardware pitch detector. IEEE Transactions on Acoustics, Speech, and Signal Processing (TASLP), pages 2-8, 1976.","title":"Real-time digital hardware pitch detector","authors":"John Dubnowski, Ronald Schafer, Lawrence Rabiner","year":"1976","venue":"IEEE Transactions on Acoustics, Speech, and Signal Processing (TASLP)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Emiya et al., 2009] Valentin Emiya, Roland Badeau, and Bertrand David. Multipitch estimation of piano sounds using a new probabilistic spectral smoothness principle. IEEE Transactions on Audio, Speech, and Language Pro- cessing (TASLP), pages 1643-1654, 2009.","title":"Multipitch estimation of piano sounds using a new probabilistic spectral smoothness principle","authors":"Valentin Emiya, Roland Badeau, Bertrand David","year":"2009","venue":"IEEE Transactions on Audio, Speech, and Language Processing (TASLP)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gardner et al., 2021] Josh Gardner, Ian Simon, Ethan Manilow, Curtis Hawthorne, and Jesse Engel. Mt3: Multi-task multitrack music transcription. arXiv preprint arXiv:2111.03017, 2021.","title":"Mt3: Multi-task multitrack music transcription","authors":"Josh Gardner, Ian Simon, Ethan Manilow, Curtis Hawthorne, Jesse Engel","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gfeller et al., 2020] Beat Gfeller, Christian Frank, Dominik Roblek, Matt Sharifi, Marco Tagliasacchi, and Mihajlo Ve- limirović. Spice: Self-supervised pitch estimation. IEEE Transactions on Audio, Speech, and Language Processing (TASLP), pages 1118-1128, 2020.","title":"Spice: Self-supervised pitch estimation","authors":"Gfeller","year":"2020","venue":"IEEE Transactions on Audio, Speech, and Language Processing (TASLP)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Guo et al., 2018] Michelle Guo, Albert Haque, De-An Huang, Serena Yeung, and Li Fei-Fei. Dynamic task pri- oritization for multitask learning. In Proceedings of the European Conference on Computer Vision (ECCV), pages 270-287, 2018.","title":"Dynamic task prioritization for multitask learning","authors":"Michelle Guo, Albert Haque, De-An, Serena Huang, Li Yeung, Fei-Fei","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hawthorne et al., 2018a] Curtis Hawthorne, Erich Elsen, Jialin Song, Adam Roberts, and Ian Simon and. Onsets and frames: Dual-objective piano transcription. In Pro- ceedings of the International Society for Music Informa- tion Retrieval Conference (ISMIR), 2018.","title":"Onsets and frames: Dual-objective piano transcription","authors":"Hawthorne","year":"2018","venue":"Proceedings of the International Society for Music Information Retrieval Conference (ISMIR)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hawthorne et al., 2018b] Curtis Hawthorne, Andriy Sta- syuk, Adam Roberts, Ian Simon, Cheng-Zhi Anna Huang, Sander Dieleman, Erich Elsen, Jesse Engel, and Douglas Eck. Enabling factorized piano music modeling and gener- ation with the maestro dataset. In Proceedings of the Inter- national Conference on Learning Representations (ICLR), 2018.","title":"Enabling factorized piano music modeling and generation with the maestro dataset","authors":"Hawthorne","year":"2018","venue":"Proceedings of the International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hawthorne et al., 2021] Curtis Hawthorne, Ian Simon, Rigel Swavely, Ethan Manilow, and Jesse Engel. Sequence-to-sequence piano transcription with transform- ers. arXiv preprint arXiv:2107.09142, 2021.","title":"Sequence-to-sequence piano transcription with transformers","authors":"Curtis Hawthorne, Ian Simon, Rigel Swavely, Ethan Manilow, Jesse Engel","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kelz et al., 2019] Rainer Kelz, Sebastian Böck, and Ger- hard Widmer. Deep polyphonic adsr piano note transcrip- tion. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 246-250, 2019.","title":"Deep polyphonic adsr piano note transcription","authors":"Rainer Kelz, Sebastian Böck, Gerhard Widmer","year":"2019","venue":"Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kim and Bello, 2019] Jong Wook Kim and Juan Pablo Bello. Adversarial learning for improved onsets and frames music transcription. arXiv preprint arXiv:1906.08512, 2019.","title":"Adversarial learning for improved onsets and frames music transcription","authors":"Bello ; ] Kim, Wook Jong, Juan Pablo Kim, Bello","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kim et al., 2018] Jong Wook Kim, Justin Salamon, Peter Li, and Juan Pablo Bello. Crepe: A convolutional repre- sentation for pitch estimation. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 161-165, 2018.","title":"Crepe: A convolutional representation for pitch estimation","authors":"Jong Wook, Kim, Justin Salamon, Peter Li, Juan Pablo Bello","year":"2018","venue":"Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kong et al., 2021] Qiuqiang Kong, Yin Cao, Haohe Liu, Keunwoo Choi, and Yuxuan Wang. Decoupling magnitude and phase estimation with deep resunet for music source separation. arXiv preprint arXiv:2109.05418, 2021.","title":"Decoupling magnitude and phase estimation with deep resunet for music source separation","authors":"Kong","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kwon et al., 2020] Taegyun Kwon, Dasaem Jeong, and Juhan Nam. Polyphonic piano transcription using au- toregressive multi-state note model. arXiv preprint arXiv:2010.01104, 2020.","title":"Polyphonic piano transcription using autoregressive multi-state note model","authors":"Kwon","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lin et al., 2017] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense ob- ject detection. In Proceedings of the IEEE international conference on computer vision (ICCV), pages 2980-2988, 2017.","title":"Focal loss for dense object detection","authors":"Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, Piotr Dollár","year":"2017","venue":"Proceedings of the IEEE international conference on computer vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lin et al., 2019] Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qing- Fu Zhang, and Sam Kwong. Pareto multi-task learn- ing. Advances in neural information processing systems (NIPS), 2019.","title":"Pareto multi-task learning","authors":"Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qing-Fu Zhang, Sam Kwong","year":"2019","venue":"Advances in neural information processing systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Liu et al., 2019] Shikun Liu, Edward Johns, and Andrew J Davison. End-to-end multi-task learning with attention. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 1871-1880, 2019.","title":"End-to-end multi-task learning with attention","authors":"Shikun Liu, Edward Johns, Andrew J Davison","year":"2019","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Lu et al., 2021] Wei-Tsung Lu, Ju-Chiang Wang, Minz Won, Keunwoo Choi, and Xuchen Song. Spectnt: a time- frequency transformer for music audio. arXiv preprint arXiv:2110.09127, 2021.","title":"Spectnt: a timefrequency transformer for music audio","authors":"Wei-Tsung Lu, Ju-Chiang Wang, Minz Won, Keunwoo Choi, Xuchen Song","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Mauch and Dixon, 2014] Matthias Mauch and Simon Dixon. pyin: A fundamental frequency estimator using probabilistic threshold distributions. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 659-663, 2014.","title":"pyin: A fundamental frequency estimator using probabilistic threshold distributions","authors":"Dixon Mauch, Matthias Mauch, Simon Dixon","year":"2014","venue":"Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[McFee et al., 2015] Brian McFee, Colin Raffel, Dawen Liang, Daniel P Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto. librosa: Audio and music signal analysis in python. In Proceedings of the 14th python in science con- ference, pages 18-25, 2015.","title":"librosa: Audio and music signal analysis in python","authors":"Brian Mcfee, Colin Raffel, Dawen Liang, P Daniel, Matt Ellis, Eric Mcvicar, Oriol Battenberg, Nieto","year":"2015","venue":"Proceedings of the 14th python in science conference","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Noll, 1967] A Michael Noll. Cepstrum pitch determination. The journal of the Acoustical Society of America, pages 293-309, 1967.","title":"Cepstrum pitch determination","authors":"Noll Michael","year":"1967","venue":"The journal of the Acoustical Society of America","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Raffel et al., 2014] Colin Raffel, Brian McFee, Eric J Humphrey, Justin Salamon, Oriol Nieto, Dawen Liang, Daniel PW Ellis, and C Colin Raffel. Mir eval: A trans- parent implementation of common mir metrics. In Pro- ceedings of the International Society for Music Informa- tion Retrieval Conference (ISMIR), pages 367-372, 2014.","title":"Mir eval: A transparent implementation of common mir metrics","authors":"Colin Raffel, Brian Mcfee, Eric J Humphrey, Justin Salamon, Oriol Nieto, Dawen Liang, Daniel Pw Ellis, Colin Raffel","year":"2014","venue":"Proceedings of the International Society for Music Information Retrieval Conference (ISMIR)","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Salamon et al., 2017] Justin Salamon, Rachel M Bittner, Jordi Bonada, Juan J Bosch, Emilia Gómez Gutiérrez, and Juan Pablo Bello. An analysis/synthesis framework for au- tomatic f0 annotation of multitrack datasets. In Proceed- ings of the International Society for Music Information Re- trieval Conference (ISMIR), 2017.","title":"An analysis/synthesis framework for automatic f0 annotation of multitrack datasets","authors":"Justin Salamon, Rachel M Bittner, Jordi Bonada, Juan J Bosch, Emilia Gómez Gutiérrez, Juan Pablo Bello","year":"2017","venue":"Proceedings of the International Society for Music Information Retrieval Conference (ISMIR)","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Su et al., 2021] Yixin Su, Rui Zhang, Sarah Erfani, and Zhenghua Xu. Detecting beneficial feature interactions for recommender systems. In Proceedings of the AAAI confer- ence on artificial intelligence, pages 4357-4365, 2021.","title":"Detecting beneficial feature interactions for recommender systems","authors":"Yixin Su, Rui Zhang, Sarah Erfani, Zhenghua Xu","year":"2021","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang and Jang, 2021] Jun-You Wang and Jyh-Shing Roger Jang. On the preparation and validation of a large-scale dataset of singing transcription. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 276-280, 2021.","title":"Jun-You Wang and Jyh-Shing Roger Jang. On the preparation and validation of a large-scale dataset of singing transcription","authors":"Jang Wang","year":"2021","venue":"Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wang et al., 2021] Xiaojie Wang, Rui Zhang, Yu Sun, and Jianzhong Qi. Combating selection biases in recommender systems with a few unbiased ratings. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, pages 427-435, 2021.","title":"Combating selection biases in recommender systems with a few unbiased ratings","authors":"Xiaojie Wang, Rui Zhang, Yu Sun, Jianzhong Qi","year":"2021","venue":"Proceedings of the 14th ACM International Conference on Web Search and Data Mining","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}