{"type":"ScholarlyArticle","title":"EditSinger: Zero-Shot Text-Based Singing Voice Editing System with Diverse Prosody Modeling","authors":[{"name":"Lichao Zhang","slug":"lichao-zhang","given":"Lichao","family":"Zhang","affiliation":"Zhejiang University"},{"name":"Zhou Zhao","slug":"zhou-zhao","given":"Zhou","family":"Zhao","affiliation":"Zhejiang University"},{"name":"Yi Ren","slug":"yi-ren","given":"Yi","family":"Ren","affiliation":"Zhejiang University"},{"name":"Liqun Deng","slug":"liqun-deng","given":"Liqun","family":"Deng","affiliation":"Huawei Noah's Ark Lab {zju zlc ,"}],"abstract":"Zero-shot text-based singing editing enables singing voice modification based on the given edited lyrics without any additional data from the target singer. However, due to the different demands, challenges occur when applying existing speech editing methods to singing voice editing task, mainly including the lack of systematic consideration concerning prosody in insertion and deletion, as well as the trade-off between the naturalness of pronunciation and the preservation of prosody in replacement. In this paper we propose EditSinger, which is a novel singing voice editing model with specially designed diverse prosody modules to overcome the challenges above. Specifically, 1) a general masked variance adaptor is introduced for the comprehensive prosody modeling of the inserted lyrics and the transition of deletion boundary; and 2) we further design a fusion pitch predictor for replacement. By disentangling the reference pitch and fusing the predicted pronunciation, the edited pitch can be reconstructed, which could ensure a natural pronunciation while preserving the prosody of the original audio. In addition, to the best of our knowledge, it is the first zero-shot text-based singing voice editing system. Our experiments conducted on the OpenSinger prove that EditSinger can synthesize high-quality edited singing voices with natural prosody according to the corresponding operations.","doi":"10.24963/ijcai.2022/625","url":"https://ijcai.org/proceedings/2022/625","pdf":"https://ijcai.org/proceedings/2022/0625.pdf","markdown":"https://ijcai.org/proceedings/2022/625.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/625","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Natural Language Processing","track":"Main Track","number":625,"pages":{"first":4503,"last":4509},"keywords":["Natural Language Processing: Applications","Natural Language Processing: Speech"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986975","length_sec":82,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985118","length_sec":361,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Chen et al., 2020] Jiawei Chen, Xu Tan, Jian Luan, Tao Qin, and Tie-Yan Liu. Hifisinger: Towards high-fidelity neural singing voice synthesis. arXiv preprint arXiv:2009.01776, 2020.","title":"Hifisinger: Towards high-fidelity neural singing voice synthesis","authors":"Jiawei Chen, Xu Tan, Jian Luan, Tao Qin, Tie-Yan Liu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Gu et al., 2021] Yu Gu, Xiang Yin, Yonghui Rao, Yuan Wan, Benlai Tang, Yang Zhang, Jitong Chen, Yuxuan Wang, and Zejun Ma. Bytesing: A chinese singing voice synthesis system using duration allocated encoder-decoder acoustic models and wavernn vocoders. In 2021 12th In- ternational Symposium on Chinese Spoken Language Pro- cessing (ISCSLP), pages 1-5. IEEE, 2021.","title":"Bytesing: A chinese singing voice synthesis system using duration allocated encoder-decoder acoustic models and wavernn vocoders","authors":"Yu Gu, Xiang Yin, Yonghui Rao, Yuan Wan, Benlai Tang, Yang Zhang, Jitong Chen, Yuxuan Wang, Zejun Ma","year":"2021","venue":"12th International Symposium on Chinese Spoken Language Processing (ISCSLP)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Huang et al., 2021] Rongjie Huang, Feiyang Chen, Yi Ren, Jinglin Liu, Chenye Cui, and Zhou Zhao. Multi-singer: Fast multi-singer singing voice vocoder with a large-scale corpus. In Proceedings of the 29th ACM International Conference on Multimedia, pages 3945-3954, 2021.","title":"Multi-singer: Fast multi-singer singing voice vocoder with a large-scale corpus","authors":"Huang","year":"2021","venue":"Proceedings of the 29th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Jin et al., 2016] Zeyu Jin, Adam Finkelstein, Stephen Di- Verdi, Jingwan Lu, and Gautham J Mysore. Cute: A con- catenative method for voice conversion using exemplar- based unit selection. In 2016 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP), pages 5660-5664. IEEE, 2016.","title":"Cute: A concatenative method for voice conversion using exemplarbased unit selection","authors":"Jin","year":"2016","venue":"IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Jin et al., 2017] Zeyu Jin, Gautham J Mysore, Stephen Di- verdi, Jingwan Lu, and Adam Finkelstein. Voco: Text- based insertion and replacement in audio narration. ACM Transactions on Graphics (TOG), 36(4):1-13, 2017.","title":"Voco: Textbased insertion and replacement in audio narration","authors":"Jin","year":"2017","venue":"ACM Transactions on Graphics (TOG)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Li et al., 2019] Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. Neural speech synthesis with trans- former network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6706-6713, 2019.","title":"Neural speech synthesis with transformer network","authors":"Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, Ming Liu","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lu et al., 2020] Peiling Lu, Jie Wu, Jian Luan, Xu Tan, and Li Zhou. Xiaoicesing: A high-quality and inte- grated singing voice synthesis system. arXiv preprint arXiv:2006.06261, 2020.","title":"Xiaoicesing: A high-quality and integrated singing voice synthesis system","authors":"Peiling Lu, Jie Wu, Jian Luan, Xu Tan, Li Zhou","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[McAuliffe et al., 2017] Michael McAuliffe, Michaela So- colof, Sarah Mihuc, Michael Wagner, and Morgan Son- deregger. Montreal forced aligner: Trainable text-speech alignment using kaldi. In Interspeech, volume 2017, pages 498-502, 2017.","title":"Montreal forced aligner: Trainable text-speech alignment using kaldi","authors":"Michael Mcauliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, Morgan Sonderegger","year":"2017","venue":"Interspeech","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Morrison et al., 2021] Max Morrison, Lucas Rencker, Zeyu Jin, Nicholas J Bryan, Juan-Pablo Caceres, and Bryan Pardo. Context-aware prosody correction for text-based speech editing. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7038-7042. IEEE, 2021.","title":"Context-aware prosody correction for text-based speech editing","authors":"Morrison","year":"2021","venue":"ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ren et al., 2019] Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech: fast, robust and controllable text to speech. In Proceedings of the 33rd International Conference on Neural Informa- tion Processing Systems, pages 3171-3180, 2019.","title":"Fastspeech: fast, robust and controllable text to speech","authors":"Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, Tie-Yan Liu","year":"2019","venue":"Proceedings of the 33rd International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ren et al., 2020a] Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech. arXiv preprint arXiv:2006.04558, 2020.","title":"Fastspeech 2: Fast and high-quality end-to-end text to speech","authors":"Ren","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ren et al., 2020b] Yi Ren, Xu Tan, Tao Qin, Jian Luan, Zhou Zhao, and Tie-Yan Liu. Deepsinger: Singing voice synthesis with data mined from the web. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1979-1989, 2020.","title":"Deepsinger: Singing voice synthesis with data mined from the web","authors":"Ren","year":"2020","venue":"Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Rubin et al., 2013] Steve Rubin, Floraine Berthouzoz, Gau- tham J Mysore, Wilmot Li, and Maneesh Agrawala. Content-based tools for editing audio stories. In Proceed- ings of the 26th annual ACM symposium on User interface software and technology, pages 113-122, 2013.","title":"Content-based tools for editing audio stories","authors":"Steve Rubin, Floraine Berthouzoz, J Gautham, Wilmot Mysore, Maneesh Li, Agrawala","year":"2013","venue":"Proceedings of the 26th annual ACM symposium on User interface software and technology","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Tan et al., 2021] Daxin Tan, Liqun Deng, Yu Ting Yeung, Xin Jiang, Xiao Chen, and Tan Lee. Editspeech: A text based speech editing system using partial inference and bidirectional fusion. arXiv preprint arXiv:2107.01554, 2021.","title":"Editspeech: A text based speech editing system using partial inference and bidirectional fusion","authors":"Daxin Tan, Liqun Deng, Yu Ting Yeung, Xin Jiang, Xiao Chen, Tan Lee","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Tang et al., 2021] Chuanxin Tang, Chong Luo, Zhiyuan Zhao, Dacheng Yin, Yucheng Zhao, and Wenjun Zeng. Zero-shot text-to-speech for text-based insertion in audio narration. arXiv preprint arXiv:2109.05426, 2021.","title":"Zero-shot text-to-speech for text-based insertion in audio narration","authors":"Chuanxin Tang, Chong Luo, Zhiyuan Zhao, Dacheng Yin, Yucheng Zhao, Wenjun Zeng","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[van den Oord et al., 2017] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete rep- resentation learning. In Proceedings of the 31st Interna- tional Conference on Neural Information Processing Sys- tems, pages 6309-6318, 2017.","title":"Neural discrete representation learning","authors":"Oord Van Den","year":"2017","venue":"Proceedings of the 31st International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2004] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transac- tions on image processing, 13(4):600-612, 2004.","title":"Image quality assessment: from error visibility to structural similarity","authors":"Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli","year":"2004","venue":"IEEE transactions on image processing","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2017] Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Ben- gio, et al. Tacotron: Towards end-to-end speech synthesis. arXiv preprint arXiv:1703.10135, 2017.","title":"Tacotron: Towards end-to-end speech synthesis","authors":"Yuxuan Wang, R J Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Yamamoto et al., 2020] Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Parallel wavegan: A fast waveform gen- eration model based on generative adversarial networks with multi-resolution spectrogram. In ICASSP 2020- 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6199-6203. IEEE, 2020.","title":"Parallel wavegan: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram","authors":"Ryuichi Yamamoto, Eunwoo Song, Jae-Min Kim","year":"2020","venue":"ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}