{"type":"ScholarlyArticle","title":"SyntaSpeech: Syntax-Aware Generative Adversarial Text-to-Speech","authors":[{"name":"Zhenhui Ye","slug":"zhenhui-ye","given":"Zhenhui","family":"Ye","affiliation":"College of Computer Science and Technology , Zhejiang University"},{"name":"Zhou Zhao","slug":"zhou-zhao","given":"Zhou","family":"Zhao","affiliation":"College of Computer Science and Technology , Zhejiang University"},{"name":"Yi Ren","slug":"yi-ren","given":"Yi","family":"Ren","affiliation":"College of Computer Science and Technology , Zhejiang University"},{"name":"Fei Wu","slug":"fei-wu","given":"Fei","family":"Wu","affiliation":"College of Computer Science and Technology , Zhejiang University"}],"abstract":"The recent progress in non-autoregressive text-to-speech (NAR-TTS) has made fast and high-quality speech synthesis possible. However, current NAR-TTS models usually use phoneme sequence as input and thus cannot understand the tree-structured syntactic information of the input sequence, which hurts the prosody modeling. To this end, we propose SyntaSpeech, a syntax-aware and light-weight NAR-TTS model, which integrates tree-structured syntactic information into the prosody modeling modules in PortaSpeech. Specifically, 1) We build a syntactic graph based on the dependency tree of the input sentence, then process the text encoding with a syntactic graph encoder to extract the syntactic information. 2) We incorporate the extracted syntactic encoding with PortaSpeech to improve the prosody prediction. 3) We introduce a multi-length discriminator to replace the flow-based post-net in PortaSpeech, which simplifies the training pipeline and improves the inference speed, while keeping the naturalness of the generated audio. Experiments on three datasets not only show that the tree-structured syntactic information grants SyntaSpeech the ability to synthesize better audio with expressive prosody, but also demonstrate the generalization ability of SyntaSpeech to adapt to multiple languages and multi-speaker text-to-speech. Ablation studies demonstrate the necessity of each component in SyntaSpeech.  Source code and audio samples are available at https://syntaspeech.github.io.","doi":"10.24963/ijcai.2022/620","url":"https://ijcai.org/proceedings/2022/620","pdf":"https://ijcai.org/proceedings/2022/0620.pdf","markdown":"https://ijcai.org/proceedings/2022/620.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/620","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Natural Language Processing","track":"Main Track","number":620,"pages":{"first":4468,"last":4474},"keywords":["Natural Language Processing: Speech"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38984631","length_sec":77,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38986488","length_sec":81,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Bińkowski et al., 2020] Mikołaj Bińkowski, Jeff Donahue, Sander Dieleman, Aidan Clark, Erich Elsen, Norman Casagrande, Luis C. Cobo, and Karen Simonyan. High fidelity speech synthesis with adversarial networks. In ICLR, 2020.","title":"High fidelity speech synthesis with adversarial networks","authors":"Mikołaj Bińkowski, Jeff Donahue, Sander Dieleman, Aidan Clark, Erich Elsen, Norman Casagrande, Luis C Cobo, Karen Simonyan","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2020] Jiawei Chen, Xu Tan, Jian Luan, Tao Qin, and Tie-Yan Liu. Hifisinger: Towards high-fidelity neural singing voice synthesis. arXiv preprint arXiv:2009.01776, 2020.","title":"Hifisinger: Towards high-fidelity neural singing voice synthesis","authors":"Jiawei Chen, Xu Tan, Jian Luan, Tao Qin, Tie-Yan Liu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In NAACL-HLT, 2019.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Donahue et al., 2021] Jeff Donahue, Sander Dieleman, Mikolaj Binkowski, Erich Elsen, and Karen Simonyan. End-to-end adversarial text-to-speech. In ICLR, 2021.","title":"End-to-end adversarial text-to-speech","authors":"Jeff Donahue, Sander Dieleman, Mikolaj Binkowski, Erich Elsen, Karen Simonyan","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hirschberg and Rambow, 2001] Julia Hirschberg and Owen Rambow. Learning prosodic features using a tree repre- sentation. In ECSCT, 2001.","title":"Learning prosodic features using a tree representation","authors":"Rambow Hirschberg, Julia Hirschberg, Owen Rambow","year":"2001","venue":"ECSCT","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ito and Johnson, 2017] Keith Ito and Linda Johnson. The lj speech dataset. https://keithito.com/LJ-Speech-Dataset/, 2017. Accessed: 2022-06-07.","title":"The lj speech dataset","authors":"Keith Ito, Linda Johnson","year":"2017","venue":null,"doi":null,"url":"https://keithito.com/LJ-Speech-Dataset/","internal_url":null},{"position":7,"raw":"[Kim et al., 2020] Jaehyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-tts: A generative flow for text-to-speech via monotonic alignment search. In NIPS, 2020.","title":"Glow-tts: A generative flow for text-to-speech via monotonic alignment search","authors":"Kim","year":"2020","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kim et al., 2021] Jaehyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with adversar- ial for end-to-end text-to-speech. In ICML, 2021.","title":"Conditional variational autoencoder with adversarial for end-to-end text-to-speech","authors":"Kim","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kong et al., 2020] Jungil Kong, Jaehyeon Kim, and Jaeky- oung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. In NIPS, 2020.","title":"Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis","authors":"Kong","year":"2020","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2016] Yujia Li, Richard Zemel, Marc Brockschmidt, and Daniel Tarlow. Gated graph se- quence neural networks. In ICLR, 2016.","title":"Gated graph sequence neural networks","authors":"Yujia Li, Richard Zemel, Marc Brockschmidt, Daniel Tarlow","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2021] Rui Liu, Berrak Sisman, and Haizhou Li. Graphspeech: Syntax-aware graph attention network for neural speech synthesis. In ICASSP, 2021.","title":"Graphspeech: Syntax-aware graph attention network for neural speech synthesis","authors":"Rui Liu, Berrak Sisman, Haizhou Li","year":"2021","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mishra et al., 2015] Taniya Mishra, Yeon-jun Kim, and Srinivas Bangalore. Intonational phrase break prediction for text-to-speech synthesis using dependency relations. In ICASSP, 2015.","title":"Intonational phrase break prediction for text-to-speech synthesis using dependency relations","authors":"Taniya Mishra, Yeon-Jun Kim, Srinivas Bangalore","year":"2015","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Peng et al., 2020] Kainan Peng, Wei Ping, Zhao Song, and Kexin Zhao. Non-autoregressive neural text-to-speech. In ICML, 2020.","title":"Non-autoregressive neural text-to-speech","authors":"Kainan Peng, Wei Ping, Zhao Song, Kexin Zhao","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ping et al., 2018] Wei Ping, Kainan Peng, Andrew Gib- iansky, Sercan O Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller. Deep voice 3: Scal- ing text-to-speech with convolutional sequence learning. In ICLR, 2018.","title":"Deep voice 3: Scaling text-to-speech with convolutional sequence learning","authors":"Wei Ping, Kainan Peng, Andrew Gibiansky, O Sercan, Ajay Arik, Sharan Kannan, Jonathan Narang, John Raiman, Miller","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ren et al., 2019] Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech: fast, robust and controllable text to speech. In NIPS, pages 3171-3180, 2019.","title":"Fastspeech: fast, robust and controllable text to speech","authors":"Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, Tie-Yan Liu","year":"2019","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ren et al., 2021a] Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech. In ICLR, 2021.","title":"Fastspeech 2: Fast and high-quality end-to-end text to speech","authors":"Ren","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ren et al., 2021b] Yi Ren, Jinglin Liu, and Zhou Zhao. Por- taspeech: Portable and high-quality generative text-to- speech. In NIPS, 2021.","title":"Portaspeech: Portable and high-quality generative text-tospeech","authors":"Ren","year":"2021","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sun et al., 2020] Aolan Sun, Jianzong Wang, Ning Cheng, Huayi Peng, Zhen Zeng, and Jing Xiao. Graphtts: graph-to-sequence modelling in neural text-to-speech. In ICASSP, 2020.","title":"Graphtts: graph-to-sequence modelling in neural text-to-speech","authors":"Sun","year":"2020","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sun et al., 2021] Aolan Sun, Jianzong Wang, Ning Cheng, Huayi Peng, Zhen Zeng, Lingwei Kong, and Jing Xiao. Graphpb: Graphical representations of prosody boundary in speech synthesis. In SLT, 2021.","title":"Graphpb: Graphical representations of prosody boundary in speech synthesis","authors":"Sun","year":"2021","venue":"SLT","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[van den Oord et al., 2016] Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Ko- ray Kavukcuoglu. WaveNet: A Generative Model for Raw Audio. In SSW, 2016.","title":"WaveNet: A Generative Model for Raw Audio","authors":"Oord Van Den","year":"2016","venue":"SSW","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2017] Yuxuan Wang, R. J. Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J. Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Z. Chen, Samy Bengio, Quoc V. Le, Yannis Agiomyrgiannakis, Robert A. J. Clark, and Rif A. Saurous. Tacotron: Towards end-to-end speech synthesis. In INTERSPEECH, 2017.","title":"Tacotron: Towards end-to-end speech synthesis","authors":"Yuxuan Wang, R J Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Z Chen, Samy Bengio, Quoc V Le, Yannis Agiomyrgiannakis, A J Robert, Rif A Clark, Saurous","year":"2017","venue":"INTERSPEECH","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yamamoto et al., 2020] Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Parallel wavegan: A fast waveform gen- eration model based on generative adversarial networks with multi-resolution spectrogram. In ICASSP, 2020.","title":"Parallel wavegan: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram","authors":"Ryuichi Yamamoto, Eunwoo Song, Jae-Min Kim","year":"2020","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zen et al., 2019] Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. Libritts: A corpus derived from librispeech for text-to-speech. arXiv preprint arXiv:1904.02882, 2019.","title":"Libritts: A corpus derived from librispeech for text-to-speech","authors":"Zen","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhou et al., 2021] Yixuan Zhou, Changhe Song, Jingbei Li, Zhiyong Wu, and Helen Meng. Dependency parsing based semantic representation learning with graph neural network for enhancing expressiveness of text-to-speech. arXiv preprint arXiv:2104.06835, 2021.","title":"Dependency parsing based semantic representation learning with graph neural network for enhancing expressiveness of text-to-speech","authors":"Yixuan Zhou, Changhe Song, Jingbei Li, Zhiyong Wu, Helen Meng","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}