{"type":"ScholarlyArticle","title":"Robust Front-End for Multi-Channel ASR using Flow-Based Density Estimation","authors":[{"name":"Hyeongju Kim","slug":"hyeongju-kim","given":"Hyeongju","family":"Kim","affiliation":"Department of Electrical and Computer Engineering and INMC , Seoul National University , South Korea"},{"name":"Hyeonseung Lee","slug":"hyeonseung-lee","given":"Hyeonseung","family":"Lee","affiliation":"Department of Electrical and Computer Engineering and INMC , Seoul National University , South Korea"},{"name":"Woo Hyun Kang","slug":"woo-hyun-kang","given":"Woo Hyun","family":"Kang","affiliation":"Department of Electrical and Computer Engineering and INMC , Seoul National University , South Korea"},{"name":"Hyung Yong Kim","slug":"hyung-yong-kim","given":"Hyung Yong","family":"Kim","affiliation":"Department of Electrical and Computer Engineering and INMC , Seoul National University , South Korea"},{"name":"Nam Soo Kim","slug":"nam-soo-kim","given":"Nam Soo","family":"Kim","affiliation":"Department of Electrical and Computer Engineering and INMC , Seoul National University , South Korea"}],"abstract":"For multi-channel speech recognition, speech enhancement techniques such as denoising or dereverberation are conventionally applied as a front-end processor. Deep learning-based front-ends using such techniques require aligned clean and noisy speech pairs which are generally obtained via data simulation. Recently, several joint optimization techniques have been proposed to train the front-end without parallel data within an end-to-end automatic speech recognition (ASR) scheme. However, the ASR objective is sub-optimal and insufficient for fully training the front-end, which still leaves room for improvement. In this paper, we propose a novel approach which incorporates flow-based density estimation for the robust front-end using non-parallel clean and noisy speech. Experimental results on the CHiME-4 dataset show that the proposed method outperforms the conventional techniques where the front-end is trained only with ASR objective.","doi":"10.24963/ijcai.2020/518","url":"https://ijcai.org/proceedings/2020/518","pdf":"https://ijcai.org/proceedings/2020/0518.pdf","markdown":"https://ijcai.org/proceedings/2020/518.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/518","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Natural Language Processing","track":"Main track","number":518,"pages":{"first":3744,"last":3750},"keywords":["Machine Learning: Deep Generative Models","Machine Learning: Transfer, Adaptation, Multi-task Learning","Natural Language Processing: Speech"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24330/file_24330.m3u8","length_sec":283,"provider":"hls"}],"references":[{"position":1,"raw":"[Barker et al., 2017] Jon Barker, Ricard Marxer, Emmanuel Vincent, and Shinji Watanabe. The third 'chime'speech separation and recognition challenge: Analysis and out- comes. Computer Speech & Language, 46:605-626, 2017.","title":"The third 'chime'speech separation and recognition challenge: Analysis and outcomes","authors":"Jon Barker, Ricard Marxer, Emmanuel Vincent, Shinji Watanabe","year":"2017","venue":"Computer Speech & Language","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Dinh et al., 2016] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.08803, 2016.","title":"Density estimation using real nvp","authors":"Laurent Dinh, Jascha Sohl-Dickstein, Samy Bengio","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Erdogan et al., 2016] Hakan Erdogan, John R Hershey, Shinji Watanabe, Michael I Mandel, and Jonathan Le Roux. Improved mvdr beamforming using single- channel mask prediction networks. In Interspeech, pages 1981-1985, 2016.","title":"Improved mvdr beamforming using singlechannel mask prediction networks","authors":"Hakan Erdogan, Shinji John R Hershey, Michael I Watanabe, Jonathan Le Mandel, Roux","year":"2016","venue":"Interspeech","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gong, 1995] Yifan Gong. Speech recognition in noisy envi- ronments: A survey. Speech communication, 16(3):261- 291, 1995.","title":"Speech recognition in noisy environments: A survey","authors":"Yifan Gong, Gong","year":"1995","venue":"Speech communication","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Han et al., 2015] Kun Han, Yuxuan Wang, DeLiang Wang, William S Woods, Ivo Merks, and Tao Zhang. Learning spectral mapping for speech dereverberation and denois- ing. IEEE/ACM Transactions on Audio, Speech, and Lan- guage Processing, 23(6):982-992, 2015.","title":"Learning spectral mapping for speech dereverberation and denoising","authors":"Kun Han, Yuxuan Wang, Deliang Wang, William S Woods, Ivo Merks, Tao Zhang","year":"2015","venue":"IEEE/ACM Transactions on Audio, Speech, and Language Processing","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Heymann et al., 2019] Jahn Heymann, Lukas Drude, Rein- hold Haeb-Umbach, Keisuke Kinoshita, and Tomohiro Nakatani. Joint optimization of neural network-based wpe dereverberation and acoustic model for robust online asr. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6655-6659. IEEE, 2019.","title":"Joint optimization of neural network-based wpe dereverberation and acoustic model for robust online asr","authors":"Jahn Heymann, Lukas Drude, Reinhold Haeb-Umbach, Keisuke Kinoshita, Tomohiro Nakatani","year":"2019","venue":"ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jensen and Taal, 2016] Jesper Jensen and Cees H Taal. An algorithm for predicting the intelligibility of speech masked by modulated noise maskers. IEEE/ACM Trans- actions on Audio, Speech, and Language Processing, 24(11):2009-2022, 2016.","title":"An algorithm for predicting the intelligibility of speech masked by modulated noise maskers","authors":"Taal ; ] Jesper Jensen, Cees H Jensen, Taal","year":"2016","venue":"IEEE/ACM Transactions on Audio, Speech, and Language Processing","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kim et al., 2017] Suyoun Kim, Takaaki Hori, and Shinji Watanabe. Joint ctc-attention based end-to-end speech recognition using multi-task learning. In 2017 IEEE inter- national conference on acoustics, speech and signal pro- cessing (ICASSP), pages 4835-4839. IEEE, 2017.","title":"Joint ctc-attention based end-to-end speech recognition using multi-task learning","authors":"Kim","year":"2017","venue":"IEEE international conference on acoustics, speech and signal processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kinoshita et al., 2016] Keisuke Kinoshita, Marc Delcroix, Sharon Gannot, Emanuël AP Habets, Reinhold Haeb- Umbach, Walter Kellermann, Volker Leutnant, Roland Maas, Tomohiro Nakatani, Bhiksha Raj, et al. A sum- mary of the reverb challenge: state-of-the-art and re- maining challenges in reverberant speech processing re- search. EURASIP Journal on Advances in Signal Process- ing, 2016(1):7, 2016.","title":"A summary of the reverb challenge: state-of-the-art and remaining challenges in reverberant speech processing research","authors":"Keisuke Kinoshita, Marc Delcroix, Sharon Gannot, A P Emanuël, Reinhold Habets, Walter Haeb-Umbach, Kellermann, Roland Volker Leutnant, Tomohiro Maas, Bhiksha Nakatani, Raj","year":"2016","venue":"EURASIP Journal on Advances in Signal Processing","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kinoshita et al., 2017] Keisuke Kinoshita, Marc Delcroix, Haeyong Kwon, Takuma Mori, and Tomohiro Nakatani. Neural network-based spectrum estimation for online wpe dereverberation. In Interspeech, pages 384-388, 2017.","title":"Neural network-based spectrum estimation for online wpe dereverberation","authors":"Keisuke Kinoshita, Marc Delcroix, Haeyong Kwon, Takuma Mori, Tomohiro Nakatani","year":"2017","venue":"Interspeech","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Meng et al., 2017] Zhong Meng, Shinji Watanabe, John R Hershey, and Hakan Erdogan. Deep long short-term mem- ory adaptive beamforming networks for multichannel ro- bust speech recognition. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 271-275. IEEE, 2017.","title":"Deep long short-term memory adaptive beamforming networks for multichannel robust speech recognition","authors":"Zhong Meng, Shinji Watanabe, John R Hershey, Hakan Erdogan","year":"2017","venue":"2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ochiai et al., 2017] Tsubasa Ochiai, Shinji Watanabe, Takaaki Hori, and John R Hershey. Multichannel end- to-end speech recognition. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 2632-2641. JMLR. org, 2017.","title":"Multichannel endto-end speech recognition","authors":"Ochiai","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Oord et al., 2016] Aaron van den Oord, Sander Diele- man, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw au- dio. arXiv preprint arXiv:1609.03499, 2016.","title":"Wavenet: A generative model for raw audio","authors":"Oord","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ping et al., 2018] Wei Ping, Kainan Peng, and Jitong Chen. Clarinet: Parallel wave generation in end-to-end text-to- speech. arXiv preprint arXiv:1807.07281, 2018.","title":"Clarinet: Parallel wave generation in end-to-end text-tospeech","authors":"Wei Ping, Kainan Peng, Jitong Chen","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Prenger et al., 2019] Ryan Prenger, Rafael Valle, and Bryan Catanzaro. Waveglow: A flow-based generative network for speech synthesis. In ICASSP 2019-2019 IEEE Inter- national Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP), pages 3617-3621. IEEE, 2019.","title":"Waveglow: A flow-based generative network for speech synthesis","authors":"Ryan Prenger, Rafael Valle, Bryan Catanzaro","year":"2019","venue":"ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Rix et al., 2001] Antony W Rix, John G Beerends, Michael P Hollier, and Andries P Hekstra. Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs. In 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No. 01CH37221), volume 2, pages 749-752. IEEE, 2001.","title":"Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs","authors":"Antony W Rix, John G Beerends, Michael P Hollier, Andries P Hekstra","year":"2001","venue":"IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No. 01CH37221)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Shen et al., 2018] Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4779-4783. IEEE, 2018.","title":"Natural tts synthesis by conditioning wavenet on mel spectrogram predictions","authors":"Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan","year":"2018","venue":"IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Vincent et al., 2006] Emmanuel Vincent, Rémi Gribonval, and Cédric Févotte. Performance measurement in blind audio source separation. IEEE transactions on audio, speech, and language processing, 14(4):1462-1469, 2006.","title":"Performance measurement in blind audio source separation","authors":"Emmanuel Vincent, Rémi Gribonval, Cédric Févotte","year":"2006","venue":"IEEE transactions on audio, speech, and language processing","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Vincent et al., 2013] Emmanuel Vincent, Jon Barker, Shinji Watanabe, Jonathan Le Roux, Francesco Nesta, and Marco Matassoni. The second 'chime'speech separation and recognition challenge: An overview of challenge systems and outcomes. In 2013 IEEE Workshop on Automatic Speech Recognition and Understanding, pages 162-167. IEEE, 2013.","title":"The second 'chime'speech separation and recognition challenge: An overview of challenge systems and outcomes","authors":"Emmanuel Vincent, Jon Barker, Shinji Watanabe, Jonathan Le Roux, Francesco Nesta, Marco Matassoni","year":"2013","venue":"IEEE Workshop on Automatic Speech Recognition and Understanding","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Watanabe et al., 2018] Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, et al. Espnet: End-to-end speech processing toolkit. arXiv preprint arXiv:1804.00015, 2018.","title":"Espnet: End-to-end speech processing toolkit","authors":"Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique, Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wölfel and McDonough, 2009] Matthias Wölfel and John W McDonough. Distant speech recognition. Wiley Online Library, 2009.","title":"Distant speech recognition","authors":"Matthias Wölfel, John W Mcdonough","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}