{"type":"ScholarlyArticle","title":"Taylor, Can You Hear Me Now? A Taylor-Unfolding Framework for Monaural Speech Enhancement","authors":[{"name":"Andong Li","slug":"andong-li","given":"Andong","family":"Li","affiliation":"Institute of Acoustics , Chinese Academy of Sciences; University of Chinese Academy of Sciences"},{"name":"Shan You","slug":"shan-you","given":"Shan","family":"You","affiliation":"SenseTime Research"},{"name":"Guochen Yu","slug":"guochen-yu","given":"Guochen","family":"Yu","affiliation":"Institute of Acoustics , Chinese Academy of Sciences; Communication University of China"},{"name":"Chengshi Zheng","slug":"chengshi-zheng","given":"Chengshi","family":"Zheng","affiliation":"Institute of Acoustics , Chinese Academy of Sciences; University of Chinese Academy of Sciences"},{"name":"Xiaodong Li","slug":"xiaodong-li","given":"Xiaodong","family":"Li","affiliation":"Institute of Acoustics , Chinese Academy of Sciences; University of Chinese Academy of Sciences"}],"abstract":"While the deep learning techniques promote the rapid development of the speech enhancement (SE) community, most schemes only pursue the performance in a black-box manner and lack adequate model interpretability. Inspired by Taylor's approximation theory, we propose an interpretable decoupling-style SE framework, which disentangles the complex spectrum recovery into two separate optimization problems i.e., magnitude and complex residual estimation. Specifically, serving as the 0th-order term in Taylor's series, a filter network is delicately devised to suppress the noise component only in the magnitude domain and obtain a coarse spectrum. To refine the phase distribution, we estimate the sparse complex residual, which is defined as the difference between target and coarse spectra, and measures the phase gap. In this study, we formulate the residual component as the combination of various high-order Taylor terms and propose a lightweight trainable module to replace the complicated derivative operator between adjacent terms. Finally, following Taylor's formula, we can reconstruct the target spectrum by the superimposition between 0th-order and high-order terms. Experimental results on two benchmark datasets show that our framework achieves state-of-the-art performance over previous competing baselines in various evaluation metrics. The source code is available at https://github.com/Andong-Li-speech/TaylorSENet.","doi":"10.24963/ijcai.2022/582","url":"https://ijcai.org/proceedings/2022/582","pdf":"https://ijcai.org/proceedings/2022/0582.pdf","markdown":"https://ijcai.org/proceedings/2022/582.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/582","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Natural Language Processing","track":"Main Track","number":582,"pages":{"first":4193,"last":4200},"keywords":["Machine Learning: Applications","Natural Language Processing: Speech","Machine Learning: Convolutional Networks"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987024","length_sec":61,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985167","length_sec":608,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Boll, 1979] Steven Boll. Suppression of acoustic noise in speech using spectral subtraction. IEEE Trans. Acoust., Speech, Signal Process., 27(2):113-120, 1979.","title":"Suppression of acoustic noise in speech using spectral subtraction","authors":"Steven Boll","year":"1979","venue":"IEEE Trans. Acoust., Speech, Signal Process","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Choi et al., 2021] Hyeong-Seok Choi, Sungjin Park, Jie Hwan Lee, Hoon Heo, Dongsuk Jeon, and Kyogu Lee. Real-Time Denoising and Dereverberation wtih Tiny Recurrent U-Net. In ICASSP, pages 5789-5793, 2021.","title":"Real-Time Denoising and Dereverberation wtih Tiny Recurrent U-Net","authors":"Hyeong-Seok Choi, Sungjin Park, Jie Hwan Lee, Hoon Heo, Dongsuk Jeon, Kyogu Lee","year":"2021","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dauphin et al., 2017] Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In ICML, pages 933-941, 2017.","title":"Language modeling with gated convolutional networks","authors":"Angela Yann N Dauphin, Michael Fan, David Auli, Grangier","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Defossez et al., 2020] Alexandre Defossez, Gabriel Syn- naeve, and Yossi Adi. Real time speech enhancement in the waveform domain. In Interspeech, pages 3291-3295, 2020.","title":"Real time speech enhancement in the waveform domain","authors":"Alexandre Defossez, Gabriel Synnaeve, Yossi Adi","year":"2020","venue":"Interspeech","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ephraim and Malah, 1984] Yariv Ephraim and David Malah. Speech enhancement using a minimum-mean square error short-time spectral amplitude estimator. IEEE Trans. Acoust., Speech, Signal Process., 32(6):1109-1121, 1984.","title":"Speech enhancement using a minimum-mean square error short-time spectral amplitude estimator","authors":"Ephraim, Malah ;, Yariv Ephraim, David Malah","year":"1984","venue":"IEEE Trans. Acoust., Speech, Signal Process","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fu et al., 2021] Xueyang Fu, Zeyu Xiao, Gang Yang, Aip- ing Liu, Zhiwei Xiong, et al. Unfolding Taylor's Approx- imations for Image Restoration. In NeurIPS, 2021.","title":"Unfolding Taylor's Approximations for Image Restoration","authors":"Fu","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gulati et al., 2020] Anmol Gulati, James Qin, Chung- Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Con- former: Convolution-augmented transformer for speech recognition. In Interspeech, pages 5036-5040, 2020.","title":"Conformer: Convolution-augmented transformer for speech recognition","authors":"Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu","year":"2020","venue":"Interspeech","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hao et al., 2021] Xiang Hao, Xiangdong Su, Radu Horaud, and Xiaofei Li. Fullsubnet: A Full-Band and Sub-Band Fusion Model for Real-Time Single-Channel Speech En- hancement. In ICASSP, pages 6633-6637, 2021.","title":"Fullsubnet: A Full-Band and Sub-Band Fusion Model for Real-Time Single-Channel Speech Enhancement","authors":"Xiang Hao, Xiangdong Su, Radu Horaud, Xiaofei Li","year":"2021","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hu et al., 2020] Yanxin Hu, Yun Liu, Shubo Lv, Mengtao Xing, Shimin Zhang, Yihui Fu, Jian Wu, Bihong Zhang, and Lei Xie. DCCRN: Deep complex convolution recur- rent network for phase-aware speech enhancement. In In- terspeech, pages 2472-2476, 2020.","title":"DCCRN: Deep complex convolution recurrent network for phase-aware speech enhancement","authors":"Yanxin Hu, Yun Liu, Shubo Lv, Mengtao Xing, Shimin Zhang, Yihui Fu, Jian Wu, Bihong Zhang, Lei Xie","year":"2020","venue":"Interspeech","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jensen and Taal, 2016] Jesper Jensen and Cees H Taal. An algorithm for predicting the intelligibility of speech masked by modulated noise maskers. IEEE/ACM Trans. Audio. Speech, Lang. Process., 24(11):2009-2022, 2016.","title":"An algorithm for predicting the intelligibility of speech masked by modulated noise maskers","authors":"Taal ; ] Jesper Jensen, Cees H Jensen, Taal","year":"2016","venue":"IEEE/ACM Trans. Audio. Speech, Lang. Process","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2021] Andong Li, Wenzhe Liu, Chengshi Zheng, Cunhang Fan, and Xiaodong Li. Two Heads are Better Than One: A Two-Stage Complex Spectral Mapping Ap- proach for Monaural Speech Enhancement. IEEE/ACM Trans. Audio. Speech, Lang. Process., 29:1829-1843, 2021.","title":"Two Heads are Better Than One: A Two-Stage Complex Spectral Mapping Approach for Monaural Speech Enhancement","authors":"Andong Li, Wenzhe Liu, Chengshi Zheng, Cunhang Fan, Xiaodong Li","year":"2021","venue":"IEEE/ACM Trans. Audio. Speech, Lang. Process","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2022] Andong Li, Chengshi Zheng, Lu Zhang, and Xiaodong Li. Glance and gaze: A collaborative learning framework for single-channel speech enhancement. Appl. Acoust., 187:108499, 2022.","title":"Glance and gaze: A collaborative learning framework for single-channel speech enhancement","authors":"Andong Li, Chengshi Zheng, Lu Zhang, Xiaodong Li","year":"2022","venue":"Appl. Acoust","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Luo and Mesgarani, 2019] Yi Luo and Nima Mesgarani. Conv-tasnet: Surpassing ideal time-frequency magnitude masking for speech separation. IEEE/ACM Trans. Audio. Speech, Lang. Process., 27(8):1256-1266, 2019.","title":"Conv-tasnet: Surpassing ideal time-frequency magnitude masking for speech separation","authors":"Yi Luo, Nima Mesgarani","year":"2019","venue":"IEEE/ACM Trans. Audio. Speech, Lang. Process","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Luo et al., 2020] Yi Luo, Zhuo Chen, and Takuya Yoshioka. Dual-path rnn: efficient long sequence modeling for time- domain single-channel speech separation. In ICASSP, pages 46-50, 2020.","title":"Dual-path rnn: efficient long sequence modeling for timedomain single-channel speech separation","authors":"Yi Luo, Zhuo Chen, Takuya Yoshioka","year":"2020","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Paliwal et al., 2011] Kuldip Paliwal, Kamil Wójcicki, and Benjamin Shannon. The importance of phase in speech enhancement. Speech. Commun., 53(4):465-494, 2011.","title":"The importance of phase in speech enhancement","authors":"Kuldip Paliwal, Kamil Wójcicki, Benjamin Shannon","year":"2011","venue":"Speech. Commun","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pandey and Wang, 2020] Ashutosh Pandey and DeLiang Wang. Densely connected neural network with dilated convolutions for real-time speech enhancement in the time domain. In ICASSP, pages 6629-6633, 2020.","title":"Densely connected neural network with dilated convolutions for real-time speech enhancement in the time domain","authors":"Wang ; Pandey, Ashutosh Pandey, Deliang Wang","year":"2020","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pascual et al., 2017] Santiago Pascual, Antonio Bonafonte, and Joan Serra. SEGAN: Speech enhancement generative adversarial network. In Interspeech, pages 3642-3646, 2017.","title":"SEGAN: Speech enhancement generative adversarial network","authors":"Pascual","year":"2017","venue":"Interspeech","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Paul and Baker, 1992] Douglas B Paul and Janet Baker. The design for the wall street journal-based csr corpus. In Workshop on Speech and Natural Lang., pages 357-362, 1992.","title":"The design for the wall street journal-based csr corpus","authors":"B Douglas, Janet Paul, Baker","year":"1992","venue":"Workshop on Speech and Natural Lang","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Qin et al., 2020] Xuebin Qin, Zichen Zhang, Chenyang Huang, Masood Dehghan, Osmar R Zaiane, and Mar- tin Jagersand. U2-Net: Going deeper with nested U- structure for salient object detection. Pattern Recognit., 106:107404, 2020.","title":"U2-Net: Going deeper with nested Ustructure for salient object detection","authors":"Qin","year":"2020","venue":"Pattern Recognit","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rec, 2005] ITUT Rec. P. 862.2: Wideband extension to recommendation p. 862 for the assessment of wideband telephone networks and speech codecs. International Telecommunication Union, CH-Geneva, 2005.","title":"ITUT Rec. P. 862.2: Wideband extension to recommendation p. 862 for the assessment of wideband telephone networks and speech codecs","authors":"Rec","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Reddy et al., 2020] Chandan KA Reddy, Vishak Gopal, Ross Culter, Ebrahim Beyrami, Roger Cheng, Harishchan- dra Dubey, Sergiy Matusevych, et al. The interspeech 2020 deep noise suppression challenge: Datasets, subjec- tive testing framework, and challenge results. In Inter- speech, pages 2492-2496, 2020.","title":"The interspeech 2020 deep noise suppression challenge: Datasets, subjective testing framework, and challenge results","authors":"K A Chandan, Vishak Reddy, Ross Gopal, Ebrahim Culter, Roger Beyrami, Harishchandra Cheng, Sergiy Dubey, Matusevych","year":"2020","venue":"Interspeech","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rix et al., 2001] Antony W Rix, John G Beerends, Michael P Hollier, and Andries P Hekstra. Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs. In ICASSP, pages 749-752, 2001.","title":"Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs","authors":"Antony W Rix, John G Beerends, Michael P Hollier, Andries P Hekstra","year":"2001","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Roux et al., 2019] Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R Hershey. SDR-half-baked or well done? In ICASSP, pages 626-630, 2019.","title":"SDR-half-baked or well done?","authors":"Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, John R Hershey","year":"2019","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Scalart and others, 1996] Pascal Scalart et al. Speech en- hancement based on a priori signal to noise estimation. In ICASSP, pages 629-632, 1996.","title":"Speech enhancement based on a priori signal to noise estimation","authors":"Pascal Scalart","year":"1996","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Taal et al., 2011] Cees H Taal, Richard C Hendriks, Richard Heusdens, and Jesper Jensen. An algorithm for intel- ligibility prediction of time-frequency weighted noisy speech. IEEE/ACM Trans. Audio. Speech, Lang. Process., 19(7):2125-2136, 2011.","title":"An algorithm for intelligibility prediction of time-frequency weighted noisy speech","authors":"Richard C Cees H Taal, Richard Hendriks, Jesper Heusdens, Jensen","year":"2011","venue":"IEEE/ACM Trans. Audio. Speech, Lang. Process","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tan and Wang, 2020] Ke Tan and DeLiang Wang. Learn- ing complex spectral mapping with gated convolutional recurrent networks for monaural speech enhancement. IEEE/ACM Trans. Audio. Speech, Lang. Process., 28:380- 390, 2020.","title":"Learning complex spectral mapping with gated convolutional recurrent networks for monaural speech enhancement","authors":"Wang ; ] Ke Tan, Deliang Tan, Wang","year":"2020","venue":"IEEE/ACM Trans. Audio. Speech, Lang. Process","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tang et al., 2021] Chuanxin Tang, Chong Luo, Zhiyuan Zhao, Wenxuan Xie, and Wenjun Zeng. Joint time- frequency and time domain learning for speech enhance- ment. In IJCAI, pages 3816-3822, 2021.","title":"Joint timefrequency and time domain learning for speech enhancement","authors":"Chuanxin Tang, Chong Luo, Zhiyuan Zhao, Wenxuan Xie, Wenjun Zeng","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2021] Zhong-Qiu Wang, Gordon Wichern, and Jonathan Le Roux. On The Compensation Between Mag- nitude and Phase in Speech Separation. IEEE Signal Pro- cess. Lett., 28:2018-2022, 2021.","title":"On The Compensation Between Magnitude and Phase in Speech Separation","authors":"Zhong-Qiu Wang, Gordon Wichern, Jonathan Le Roux","year":"2021","venue":"IEEE Signal Process. Lett","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Westhausen and Meyer, 2020] Nils L Westhausen and Bernd T Meyer. Dual-signal transformation lstm network for real-time noise suppression. In Interspeech, pages 2477-2481, 2020.","title":"Dual-signal transformation lstm network for real-time noise suppression","authors":"Meyer ; Westhausen, L Nils, Bernd T Westhausen, Meyer","year":"2020","venue":"Interspeech","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Williamson et al., 2015] Donald S Williamson, Yuxuan Wang, and DeLiang Wang. Complex ratio masking for monaural speech separation. IEEE/ACM Trans. Audio. Speech, Lang. Process., 24(3):483-492, 2015.","title":"Complex ratio masking for monaural speech separation","authors":"Yuxuan Donald S Williamson, Deliang Wang, Wang","year":"2015","venue":"IEEE/ACM Trans. Audio. Speech, Lang. Process","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yin et al., 2020] Dacheng Yin, Chong Luo, Zhiwei Xiong, and Wenjun Zeng. PHASEN: A phase-and-harmonics- aware speech enhancement network. In AAAI, pages 9458-9465, 2020.","title":"PHASEN: A phase-and-harmonicsaware speech enhancement network","authors":"Dacheng Yin, Chong Luo, Zhiwei Xiong, Wenjun Zeng","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}