{"type":"ScholarlyArticle","title":"FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis","authors":[{"name":"Rongjie Huang","slug":"rongjie-huang","given":"Rongjie","family":"Huang","affiliation":"Zhejiang University"},{"name":"Max W. Y. Lam","slug":"max-w-y-lam","given":"Max W. Y.","family":"Lam","affiliation":"Tencent AI Lab , China"},{"name":"Jun Wang","slug":"jun-wang","given":"Jun","family":"Wang","affiliation":"Tencent AI Lab , China"},{"name":"Dan Su","slug":"dan-su","given":"Dan","family":"Su","affiliation":"Tencent AI Lab , China"},{"name":"Dong Yu","slug":"dong-yu","given":"Dong","family":"Yu","affiliation":"Tencent AI Lab , USA"},{"name":"Yi Ren","slug":"yi-ren","given":"Yi","family":"Ren","affiliation":"Zhejiang University"},{"name":"Zhou Zhao","slug":"zhou-zhao","given":"Zhou","family":"Zhao","affiliation":"Zhejiang University"}],"abstract":"Denoising diffusion probabilistic models (DDPMs) have recently achieved leading performances in many generative tasks. However, the inherited iterative sampling process costs hindered their applications to speech synthesis. This paper proposes FastDiff, a fast conditional diffusion model for high-quality speech synthesis. FastDiff employs a stack of time-aware location-variable convolutions of diverse receptive field patterns to efficiently model long-term time dependencies with adaptive conditions. A noise schedule predictor is also adopted to reduce the sampling steps without sacrificing the generation quality. Based on FastDiff, we design an end-to-end text-to-speech synthesizer, FastDiff-TTS, which generates high-fidelity speech waveforms without any intermediate feature (e.g., Mel-spectrogram). Our evaluation of FastDiff demonstrates the state-of-the-art results with higher-quality (MOS 4.28) speech samples. Also, FastDiff enables a sampling speed of 58x faster than real-time on a V100 GPU, making diffusion models practically applicable to speech synthesis deployment for the first time. We further show that FastDiff generalized well to the mel-spectrogram inversion of unseen speakers, and FastDiff-TTS outperformed other competing methods in end-to-end text-to-speech synthesis. Audio samples are available at https://FastDiff.github.io/.","doi":"10.24963/ijcai.2022/577","url":"https://ijcai.org/proceedings/2022/577","pdf":"https://ijcai.org/proceedings/2022/0577.pdf","markdown":"https://ijcai.org/proceedings/2022/577.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/577","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Natural Language Processing","track":"Main Track","number":577,"pages":{"first":4157,"last":4163},"keywords":["Natural Language Processing: Language Generation","Natural Language Processing: Speech"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987000","length_sec":77,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985143","length_sec":640,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Chen et al., 2020] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. In Proc. of ICLR, 2020.","title":"Wavegrad: Estimating gradients for waveform generation","authors":"Chen","year":"2020","venue":"Proc. of ICLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2021] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, Najim Dehak, and William Chan. Wavegrad 2: Iterative refinement for text- to-speech synthesis. arXiv preprint arXiv:2106.09660, 2021.","title":"Wavegrad 2: Iterative refinement for textto-speech synthesis","authors":"Chen","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Creswell et al., 2018] Antonia Creswell, Tom White, Vin- cent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A Bharath. Generative adversarial networks: An overview. IEEE Signal Processing Magazine, 2018.","title":"Generative adversarial networks: An overview","authors":"Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, Anil A Bharath","year":"2018","venue":"IEEE Signal Processing Magazine","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dhariwal and Nichol, 2021] Prafulla Dhariwal and Alexan- der Nichol. Diffusion models beat gans on image synthe- sis. In Proc. of NeurIPS, volume 34, 2021.","title":"Diffusion models beat gans on image synthesis","authors":"Nichol Dhariwal","year":"2021","venue":"Proc. of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Donahue et al., 2020] Jeff Donahue, Sander Dieleman, Mikolaj Binkowski, Erich Elsen, and Karen Simonyan. End-to-end adversarial text-to-speech. In Proc. of ICLR, 2020.","title":"End-to-end adversarial text-to-speech","authors":"Jeff Donahue, Sander Dieleman, Mikolaj Binkowski, Erich Elsen, Karen Simonyan","year":"2020","venue":"Proc. of ICLR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ho et al., 2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Proc. of NeurIPS, 2020.","title":"Denoising diffusion probabilistic models","authors":"Jonathan Ho, Ajay Jain, Pieter Abbeel","year":"2020","venue":"Proc. of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Huang et al., 2021] Rongjie Huang, Feiyang Chen, Yi Ren, Jinglin Liu, Chenye Cui, and Zhou Zhao. Multi-singer: Fast multi-singer singing voice vocoder with a large-scale corpus. In Proc. of ACM MM, 2021.","title":"Multi-singer: Fast multi-singer singing voice vocoder with a large-scale corpus","authors":"Huang","year":"2021","venue":"Proc. of ACM MM","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ito and Johnson, 2017] Keith Ito and Linda Johnson. The lj speech dataset. https://keithito.com/LJ-Speech-Dataset/, 2017. Accessed: 2022-01-01.","title":"The lj speech dataset","authors":"Keith Ito, Linda Johnson","year":"2017","venue":null,"doi":null,"url":"https://keithito.com/LJ-Speech-Dataset/","internal_url":null},{"position":9,"raw":"[Jang et al., 2021] Won Jang, Dan Lim, Jaesam Yoon, Bong- wan Kim, and Juntae Kim. Univnet: A neural vocoder with multi-resolution spectrogram discriminators for high- fidelity waveform generation. In Proc. of InterSpeech, 2021.","title":"Univnet: A neural vocoder with multi-resolution spectrogram discriminators for highfidelity waveform generation","authors":"Jang","year":"2021","venue":"Proc. of InterSpeech","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kim et al., 2020] Jaehyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-tts: A generative flow for text-to-speech via monotonic alignment search. In Proc. of NeurIPS, volume 33, pages 8067-8077, 2020.","title":"Glow-tts: A generative flow for text-to-speech via monotonic alignment search","authors":"Kim","year":"2020","venue":"Proc. of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kim et al., 2021] Jaehyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In Proc. of ICML, pages 5530-5540. PMLR, 2021.","title":"Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech","authors":"Kim","year":"2021","venue":"Proc. of ICML","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kong et al., 2020a] Jungil Kong, Jaehyeon Kim, and Jaeky- oung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Proc. of NeurIPS, 33:17022-17033, 2020.","title":"Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis","authors":"Kong","year":"2020","venue":"Proc. of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kong et al., 2020b] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versa- tile diffusion model for audio synthesis. In Proc. of ICLR, 2020.","title":"Diffwave: A versatile diffusion model for audio synthesis","authors":"Kong","year":"2020","venue":"Proc. of ICLR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lam et al., 2022] Max WY Lam, Jun Wang, Dan Su, and Dong Yu. Bddm: Bilateral denoising diffusion models for fast and high-quality speech synthesis. In Proc. of ICLR, 2022.","title":"Bddm: Bilateral denoising diffusion models for fast and high-quality speech synthesis","authors":"Max Wy Lam, Jun Wang, Dan Su, Dong Yu","year":"2022","venue":"Proc. of ICLR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2021] Jinglin Liu, Chengxi Li, Yi Ren, Feiyang Chen, Peng Liu, and Zhou Zhao. Diffsinger: Singing voice synthesis via shallow diffusion mechanism. arXiv preprint arXiv:2105.02446, 2, 2021.","title":"Diffsinger: Singing voice synthesis via shallow diffusion mechanism","authors":"Jinglin Liu, Chengxi Li, Yi Ren, Feiyang Chen, Peng Liu, Zhou Zhao","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Oord et al., 2016] Aaron van den Oord, Sander Diele- man, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw au- dio. arXiv preprint arXiv:1609.03499, 2016.","title":"Wavenet: A generative model for raw audio","authors":"Oord","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Prenger et al., 2019] Ryan Prenger, Rafael Valle, and Bryan Catanzaro. Waveglow: A flow-based generative network for speech synthesis. In Proc. of ICASSP, 2019.","title":"Waveglow: A flow-based generative network for speech synthesis","authors":"Ryan Prenger, Rafael Valle, Bryan Catanzaro","year":"2019","venue":"Proc. of ICASSP","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ren et al., 2020] Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech. In Proc. of ICLR, 2020.","title":"Fastspeech 2: Fast and high-quality end-to-end text to speech","authors":"Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, Tie-Yan Liu","year":"2020","venue":"Proc. of ICLR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ren et al., 2021] Yi Ren, Jinglin Liu, and Zhou Zhao. Por- taspeech: Portable and high-quality generative text-to- speech. In Proc. of NeurIPS, volume 34, 2021.","title":"Portaspeech: Portable and high-quality generative text-tospeech","authors":"Yi Ren, Jinglin Liu, Zhou Zhao","year":"2021","venue":"Proc. of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rix et al., 2001] Antony W Rix, John G Beerends, Michael P Hollier, and Andries P Hekstra. Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs. In Proc. of ICASSP, 2001.","title":"Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs","authors":"Antony W Rix, John G Beerends, Michael P Hollier, Andries P Hekstra","year":"2001","venue":"Proc. of ICASSP","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Song et al., 2020] Jiaming Song, Chenlin Meng, and Ste- fano Ermon. Denoising diffusion implicit models. In Proc. of ICLR, 2020.","title":"Denoising diffusion implicit models","authors":"Jiaming Song, Chenlin Meng, Stefano Ermon","year":"2020","venue":"Proc. of ICLR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Taal et al., 2010] Cees H Taal, Richard C Hendriks, Richard Heusdens, and Jesper Jensen. A short-time objective in- telligibility measure for time-frequency weighted noisy speech. In Proc. of ICASSP, 2010.","title":"A short-time objective intelligibility measure for time-frequency weighted noisy speech","authors":"Richard C Cees H Taal, Richard Hendriks, Jesper Heusdens, Jensen","year":"2010","venue":"Proc. of ICASSP","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. of NeurIPS, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Proc. of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xiao et al., 2021] Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with de- noising diffusion gans. arXiv preprint arXiv:2112.07804, 2021.","title":"Tackling the generative learning trilemma with denoising diffusion gans","authors":"Xiao","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yamagishi et al., 2019] Junichi Yamagishi, Christophe Veaux, Kirsten MacDonald, et al. Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit (version 0.92). 2019.","title":"Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit","authors":"Junichi Yamagishi, Christophe Veaux, Kirsten Macdonald","year":"2019","venue":"version","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zeng et al., 2021] Zhen Zeng, Jianzong Wang, Ning Cheng, and Jing Xiao. Lvcnet: Efficient condition-dependent modeling network for waveform generation. In Proc. of ICASSP, 2021.","title":"Lvcnet: Efficient condition-dependent modeling network for waveform generation","authors":"Zhen Zeng, Jianzong Wang, Ning Cheng, Jing Xiao","year":"2021","venue":"Proc. of ICASSP","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}