{"type":"ScholarlyArticle","title":"Joint Time-Frequency and Time Domain Learning for Speech Enhancement","authors":[{"name":"Chuanxin Tang","slug":"chuanxin-tang","given":"Chuanxin","family":"Tang","affiliation":"Microsoft Research Asia"},{"name":"Chong Luo","slug":"chong-luo","given":"Chong","family":"Luo","affiliation":"Microsoft Research Asia"},{"name":"Zhiyuan Zhao","slug":"zhiyuan-zhao","given":"Zhiyuan","family":"Zhao","affiliation":"Microsoft Research Asia"},{"name":"Wenxuan Xie","slug":"wenxuan-xie","given":"Wenxuan","family":"Xie","affiliation":"Microsoft Research Asia"},{"name":"Wenjun Zeng","slug":"wenjun-zeng","given":"Wenjun","family":"Zeng","affiliation":"Microsoft Research Asia"}],"abstract":"For single-channel speech enhancement, both time-domain and time-frequency-domain methods have their respective pros and cons. In this paper, we present a cross-domain framework named TFT-Net, which takes time-frequency spectrogram as input and produces time-domain waveform as output. Such a framework takes advantage of the knowledge we have about spectrogram and avoids some of the drawbacks that T-F-domain methods have been suffering from. In TFT-Net, we design an innovative dual-path attention block (DAB) to fully exploit correlations along the time and frequency axes. We further discover that a sample-independent DAB (SDAB) achieves a good tradeoff between enhanced speech quality and complexity. Ablation studies show that both the cross-domain design and the SDAB block bring large performance gain. When logarithmic MSE is used as the training criteria, TFT-Net achieves the highest SDR and SSNR among state-of-the-art methods on two major speech enhancement benchmarks.","doi":"10.24963/ijcai.2020/528","url":"https://ijcai.org/proceedings/2020/528","pdf":"https://ijcai.org/proceedings/2020/0528.pdf","markdown":"https://ijcai.org/proceedings/2020/528.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/528","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Natural Language Processing","track":"Main track","number":528,"pages":{"first":3816,"last":3822},"keywords":["Machine Learning: Deep Learning","Natural Language Processing: Speech"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24705/file_24705.m3u8","length_sec":297,"provider":"hls"}],"references":[{"position":1,"raw":"[Choi et al., 2019] Hyeong-Seok Choi, Jang-Hyun Kim, Jae- sung Huh, Adrian Kim, Jung-Woo Ha, and Kyogu Lee. Phase-aware speech enhancement with deep complex u- net. arXiv:1903.03107, 2019.","title":"Phase-aware speech enhancement with deep complex unet","authors":"Hyeong-Seok Choi, Jang-Hyun Kim, Jaesung Huh, Adrian Kim, Jung-Woo Ha, Kyogu Lee","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ephrat et al., 2018] Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T Freeman, and Michael Rubinstein. Looking to listen at the cocktail party: A speaker-independent audio- visual model for speech separation. arXiv:1804.03619, 2018.","title":"Looking to listen at the cocktail party: A speaker-independent audiovisual model for speech separation","authors":"Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, Michael William T Freeman, Rubinstein","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Erdogan et al., 2015] Hakan Erdogan, John R Hershey, Shinji Watanabe, and Jonathan Le Roux. Phase-sensitive and recognition-boosted speech separation using deep re- current neural networks. In ICASSP, pages 708-712, 2015.","title":"Phase-sensitive and recognition-boosted speech separation using deep recurrent neural networks","authors":"Hakan Erdogan, Shinji John R Hershey, Jonathan Le Watanabe, Roux","year":"2015","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gemmeke et al., 2017] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Chan- ning Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In ICASSP, pages 776-780, 2017.","title":"Audio set: An ontology and human-labeled dataset for audio events","authors":"Jort F Gemmeke, P W Daniel, Dylan Ellis, Aren Freedman, Wade Jansen, R Channing Lawrence, Manoj Moore, Marvin Plakal, Ritter","year":"2017","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Germain et al., 2018] Francois G Germain, Qifeng Chen, and Vladlen Koltun. Speech denoising with deep feature losses. arXiv:1806.10522, 2018.","title":"Speech denoising with deep feature losses","authors":"Qifeng Francois G Germain, Vladlen Chen, Koltun","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hu and Loizou, 2007] Yi Hu and Philipos C Loizou. Eval- uation of objective quality measures for speech enhance- ment. TASLP, 16(1):229-238, 2007.","title":"Evaluation of objective quality measures for speech enhancement","authors":"Yi Hu, Philipos C Loizou","year":"2007","venue":"TASLP","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv:1502.03167, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Sergey Ioffe, Christian Szegedy","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kim et al., 2018] Jang-Hyun Kim, Jaejun Yoo, Sanghyuk Chun, Adrian Kim, and Jung-Woo Ha. Multi-domain pro- cessing via hybrid denoising networks for speech enhance- ment. arXiv:1812.08914, 2018.","title":"Multi-domain processing via hybrid denoising networks for speech enhancement","authors":"Jang-Hyun Kim, Jaejun Yoo, Sanghyuk Chun, Adrian Kim, Jung-Woo Ha","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kim et al., 2019] Jaeyoung Kim, Mostafa El-Khamy, and Jungwon Lee. Transformer with gaussian weighted self- attention for speech enhancement. arXiv:1910.06762, 2019.","title":"Transformer with gaussian weighted selfattention for speech enhancement","authors":"Kim","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Krawczyk and Gerkmann, 2014] Martin Krawczyk and Timo Gerkmann. Stft phase reconstruction in voiced speech for an improved single-channel speech enhance- ment. TASLP, 22(12):1931-1940, 2014.","title":"Stft phase reconstruction in voiced speech for an improved single-channel speech enhancement","authors":"Martin Krawczyk, Timo Gerkmann","year":"2014","venue":"TASLP","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Luo and Mesgarani, 2019] Yi Luo and Nima Mesgarani. Conv-tasnet: Surpassing ideal time-frequency magnitude masking for speech separation. TASLP, 27(8), 2019.","title":"Conv-tasnet: Surpassing ideal time-frequency magnitude masking for speech separation","authors":"Yi Luo, Nima Mesgarani","year":"2019","venue":"TASLP","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Paliwal et al., 2011] Kuldip Paliwal, Kamil Wójcicki, and Benjamin Shannon. The importance of phase in speech en- hancement. speech communication, 53(4):465-494, 2011.","title":"The importance of phase in speech enhancement","authors":"Kuldip Paliwal, Kamil Wójcicki, Benjamin Shannon","year":"2011","venue":"speech communication","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Pandey and Wang, 2019] Ashutosh Pandey and DeLiang Wang. Tcnn: Temporal convolutional neural network for real-time speech enhancement in the time domain. In ICASSP, pages 6875-6879, 2019.","title":"Tcnn: Temporal convolutional neural network for real-time speech enhancement in the time domain","authors":"Wang ; Pandey, Ashutosh Pandey, Deliang Wang","year":"2019","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pascual et al., 2017] Santiago Pascual, Antonio Bonafonte, and Joan Serra. Segan: Speech enhancement generative adversarial network. arXiv:1703.09452, 2017.","title":"Segan: Speech enhancement generative adversarial network","authors":"Pascual","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Plapous et al., 2005] Cyril Plapous, Claude Marro, and Pas- cal Scalart. Speech enhancement using harmonic regener- ation. In ICASSP, volume 1, pages I-157, 2005.","title":"Speech enhancement using harmonic regeneration","authors":"Cyril Plapous, Claude Marro, Pascal Scalart","year":"2005","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rethage et al., 2018] Dario Rethage, Jordi Pons, and Xavier Serra. A wavenet for speech denoising. In ICASSP, pages 5069-5073, 2018.","title":"A wavenet for speech denoising","authors":"Dario Rethage, Jordi Pons, Xavier Serra","year":"2018","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Scalart and FILHO, 1996] Pascal Scalart and Jozue VIEIRA FILHO. Speech enhancement based on a priori signal to noise estimation. In ICASSP, volume 2, pages 629-632, 1996.","title":"Speech enhancement based on a priori signal to noise estimation","authors":"Pascal Scalart, Jozue Vieira Filho","year":"1996","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Soni et al., 2018] Meet H Soni, Neil Shah, and Hemant A Patil. Time-frequency masking-based speech enhance- ment using generative adversarial network. In ICASSP, pages 5039-5043, 2018.","title":"Time-frequency masking-based speech enhancement using generative adversarial network","authors":"H Meet, Neil Soni, Hemant A Shah, Patil","year":"2018","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Thiemann et al., 2013] Joachim Thiemann, Nobutaka Ito, and Emmanuel Vincent. The diverse environments multi- channel acoustic noise database (demand): A database of multichannel environmental noise recordings. In Proceed- ings of Meetings on Acoustics ICA2013, volume 19, page 035081. ASA, 2013.","title":"The diverse environments multichannel acoustic noise database (demand): A database of multichannel environmental noise recordings","authors":"Joachim Thiemann, Nobutaka Ito, Emmanuel Vincent","year":"2013","venue":"Proceedings of Meetings on Acoustics ICA2013","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Valentini-Botinhao et al., 2016] Cassia Valentini-Botinhao, Xin Wang, Shinji Takaki, and Junichi Yamagishi. Investi- gating rnn-based speech enhancement methods for noise- robust text-to-speech. In SSW, pages 146-152, 2016.","title":"Investigating rnn-based speech enhancement methods for noiserobust text-to-speech","authors":"Valentini-Botinhao","year":"2016","venue":"SSW","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Veaux et al., 2013] Christophe Veaux, Junichi Yamagishi, and Simon King. The voice bank corpus: Design, col- lection and data analysis of a large regional accent speech database. In O-COCOSDA/CASLRE, pages 1-4, 2013.","title":"The voice bank corpus: Design, collection and data analysis of a large regional accent speech database","authors":"Christophe Veaux, Junichi Yamagishi, Simon King","year":"2013","venue":"O-COCOSDA/CASLRE","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vincent et al., 2006] Emmanuel Vincent, Rémi Gribonval, and Cédric Févotte. Performance measurement in blind audio source separation. TASLP, 14(4):1462-1469, 2006.","title":"Performance measurement in blind audio source separation","authors":"Emmanuel Vincent, Rémi Gribonval, Cédric Févotte","year":"2006","venue":"TASLP","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wakabayashi et al., 2018] Yukoh Wakabayashi, Takahiro Fukumori, Masato Nakayama, Takanobu Nishiura, and Yoichi Yamashita. Single-channel speech enhancement with phase reconstruction based on phase distortion aver- aging. TASLP, 26(9):1559-1569, 2018.","title":"Single-channel speech enhancement with phase reconstruction based on phase distortion averaging","authors":"Wakabayashi","year":"2018","venue":"TASLP","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang and Chen, 2018] DeLiang Wang and Jitong Chen. Supervised speech separation based on deep learning: An overview. TASLP, 26(10):1702-1726, 2018.","title":"Supervised speech separation based on deep learning: An overview","authors":"Wang, Chen ;, Deliang Wang, Jitong Chen","year":"2018","venue":"TASLP","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Williamson et al., 2016] Donald S Williamson, Yuxuan Wang, and DeLiang Wang. Complex ratio masking for monaural speech separation. TASLP, 24(3):483-492, 2016.","title":"Complex ratio masking for monaural speech separation","authors":"Yuxuan Donald S Williamson, Deliang Wang, Wang","year":"2016","venue":"TASLP","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yin et al., 2019] Dacheng Yin, Chong Luo, Zhiwei Xiong, and Wenjun Zeng. Phasen: A phase-and-harmonics-aware speech enhancement network. arXiv:1911.04697, 2019.","title":"Phasen: A phase-and-harmonics-aware speech enhancement network","authors":"Dacheng Yin, Chong Luo, Zhiwei Xiong, Wenjun Zeng","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}