{"type":"ScholarlyArticle","title":"Multi-Scale Group Transformer for Long Sequence Modeling in Speech Separation","authors":[{"name":"Yucheng Zhao","slug":"yucheng-zhao","given":"Yucheng","family":"Zhao","affiliation":"University of Science and Technology of China"},{"name":"Chong Luo","slug":"chong-luo","given":"Chong","family":"Luo","affiliation":"Microsoft Research Asia"},{"name":"Zheng-Jun Zha","slug":"zheng-jun-zha","given":"Zheng-Jun","family":"Zha","affiliation":"University of Science and Technology of China"},{"name":"Wenjun Zeng","slug":"wenjun-zeng","given":"Wenjun","family":"Zeng","affiliation":"Microsoft Research Asia"}],"abstract":"In this paper, we introduce Transformer to the time-domain methods for single-channel speech separation. Transformer has the potential to boost speech separation performance because of its strong sequence modeling capability. However, its computational complexity, which grows quadratically with the sequence length, has made it largely inapplicable to speech applications. To tackle this issue, we propose a novel variation of Transformer, named multi-scale group Transformer (MSGT). The key ideas are group self-attention, which significantly reduces the complexity, and multi-scale fusion, which retains Transform's ability to capture long-term dependency. We implement two versions of MSGT with different complexities, and apply them to a well-known time-domain speech separation method called Conv-TasNet. By simply replacing the original temporal convolutional network (TCN) with MSGT, our approach called MSGT-TasNet achieves a large gain over Conv-TasNet on both WSJ0-2mix and WHAM! benchmarks. Without bells and whistles, the performance of MSGT-TasNet is already on par with the SOTA methods.","doi":"10.24963/ijcai.2020/450","url":"https://ijcai.org/proceedings/2020/450","pdf":"https://ijcai.org/proceedings/2020/0450.pdf","markdown":"https://ijcai.org/proceedings/2020/450.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/450","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":450,"pages":{"first":3251,"last":3257},"keywords":["Machine Learning: Deep Learning: Sequence Modeling","Natural Language Processing: Speech"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24892/file_24892.m3u8","length_sec":300,"provider":"hls"}],"references":[{"position":1,"raw":"[Bai et al., 2018] Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271, 2018.","title":"An empirical evaluation of generic convolutional and recurrent networks for sequence modeling","authors":"Bai","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Child et al., 2019] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.","title":"Rewon Child","authors":"Child","year":"2019","venue":"Generating long sequences with sparse transformers","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dai et al., 2019] Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Rus- lan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860, 2019.","title":"Transformer-xl: Attentive language models beyond a fixed-length context","authors":"Dai","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Heitkaemper et al., 2019] Jens Heitkaemper, Darius Jakobeit, Christoph Boeddeker, Lukas Drude, and Rein- hold Haeb-Umbach. Demystifying tasnet: A dissecting approach. arXiv preprint arXiv:1911.08895, 2019.","title":"Demystifying tasnet: A dissecting approach","authors":"Heitkaemper","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hershey et al., 2016] John R Hershey, Zhuo Chen, Jonathan Le Roux, and Shinji Watanabe. Deep clustering: Discrim- inative embeddings for segmentation and separation. In ICASSP, pages 31-35, 2016.","title":"Deep clustering: Discriminative embeddings for segmentation and separation","authors":"Zhuo John R Hershey, Jonathan Le Chen, Shinji Roux, Watanabe","year":"2016","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Isik et al., 2016] Yusuf Isik, Jonathan Le Roux, Zhuo Chen, Shinji Watanabe, and John R Hershey. Single-channel multi-speaker separation using deep clustering. Inter- speech, pages 545-549, 2016.","title":"Single-channel multi-speaker separation using deep clustering","authors":"Yusuf Isik, Jonathan Le Roux, Zhuo Chen, Shinji Watanabe, John R Hershey","year":"2016","venue":"Interspeech","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kolbæk et al., 2017] Morten Kolbæk, Dong Yu, Zheng-Hua Tan, Jesper Jensen, Morten Kolbaek, Dong Yu, Zheng- Hua Tan, and Jesper Jensen. Multitalker speech sepa- ration with utterance-level permutation invariant training of deep recurrent neural networks. IEEE/ACM TASLP, 25(10):1901-1913, 2017.","title":"Multitalker speech separation with utterance-level permutation invariant training of deep recurrent neural networks","authors":"Morten Kolbæk, Dong Yu, Zheng-Hua Tan, Jesper Jensen, Morten Kolbaek, Dong Yu, Zheng-Hua Tan, Jesper Jensen","year":"2017","venue":"IEEE/ACM TASLP","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Le Roux et al., 2019] Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R Hershey. Sdr-half-baked or well done? In ICASSP, pages 626-630, 2019.","title":"Sdr-half-baked or well done?","authors":"Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, John R Hershey","year":"2019","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Liu and Lapata, 2019] Yang Liu and Mirella Lapata. Hier- archical transformers for multi-document summarization. arXiv preprint arXiv:1905.13164, 2019.","title":"Hierarchical transformers for multi-document summarization","authors":"Yang Liu, Mirella Lapata","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu and Wang, 2019] Yuzhou Liu and DeLiang Wang. Di- vide and conquer: A deep casa approach to talker- independent monaural speaker separation. arXiv preprint arXiv:1904.11148, 2019.","title":"Divide and conquer: A deep casa approach to talkerindependent monaural speaker separation","authors":"Wang ; Liu, Yuzhou Liu, Deliang Wang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2018] Hanxiao Liu, Karen Simonyan, and Yim- ing Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055, 2018.","title":"Darts: Differentiable architecture search","authors":"Liu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Luo and Mesgarani, 2018] Yi Luo and Nima Mesgarani. Tasnet: time-domain audio separation network for real- time, single-channel speech separation. In ICASSP, pages 696-700, 2018.","title":"Tasnet: time-domain audio separation network for realtime, single-channel speech separation","authors":"Yi Luo, Nima Mesgarani","year":"2018","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Luo and Mesgarani, 2019] Yi Luo and Nima Mesgarani. Conv-tasnet: Surpassing ideal time-frequency magni- tude masking for speech separation. IEEE/ACM TASLP, 27(8):1256-1266, 2019.","title":"Conv-tasnet: Surpassing ideal time-frequency magnitude masking for speech separation","authors":"Mesgarani ; ] Luo, Luo Yi, Nima Mesgarani","year":"2019","venue":"IEEE/ACM TASLP","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Miculicich et al., 2018] Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas, and James Henderson. Document-level neural machine translation with hierarchical attention net- works. In EMNLP, pages 2947-2954, 2018.","title":"Document-level neural machine translation with hierarchical attention networks","authors":"Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas, James Henderson","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Rix et al., 2001] Antony W Rix, John G Beerends, Michael P Hollier, and Andries P Hekstra. Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs. In ICASSP, pages 749-752, 2001.","title":"Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs","authors":"Antony W Rix, John G Beerends, Michael P Hollier, Andries P Hekstra","year":"2001","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ronneberger et al., 2015] Olaf Ronneberger, Philipp Fis- cher, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.","title":"Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation","authors":"Ronneberger","year":"2015","venue":"MICCAI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Shen et al., 2018] Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, and Chengqi Zhang. Bi-directional block self- attention for fast and memory-efficient sequence model- ing. In ICLR, 2018.","title":"Bi-directional block selfattention for fast and memory-efficient sequence modeling","authors":"Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Chengqi Zhang","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Shi et al., 2018] Jing Shi, Jiaming Xu, Guangcan Liu, and Bo Xu. Listen, think and listen again: Capturing top-down auditory attention for speaker-independent speech separa- tion. In IJCAI, pages 4353-4360, 2018.","title":"Listen, think and listen again: Capturing top-down auditory attention for speaker-independent speech separation","authors":"Jing Shi, Jiaming Xu, Guangcan Liu, Bo Xu","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/605"},{"position":20,"raw":"[Sun et al., 2019] Ke Sun, Bin Xiao, Dong Liu, and Jing- dong Wang. Deep high-resolution representation learning for human pose estimation. In CVPR, 2019.","title":"Deep high-resolution representation learning for human pose estimation","authors":"Ke Sun, Bin Xiao, Dong Liu, Jingdong Wang","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Vincent et al., 2006] Emmanuel Vincent, Rémi Gribonval, and Cédric Févotte. Performance measurement in blind audio source separation. TASLP, 14(4):1462-1469, 2006.","title":"Performance measurement in blind audio source separation","authors":"Emmanuel Vincent, Rémi Gribonval, Cédric Févotte","year":"2006","venue":"TASLP","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang and Chen, 2018] DeLiang Wang and Jitong Chen. Supervised speech separation based on deep learning: An overview. IEEE/ACM TASLP, 26(10):1702-1726, 2018.","title":"Supervised speech separation based on deep learning: An overview","authors":"Wang, Chen ;, Deliang Wang, Jitong Chen","year":"2018","venue":"IEEE/ACM TASLP","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2018] Zhong-Qiu Wang, Jonathan Le Roux, DeLiang Wang, and John R Hershey. End-to-end speech separation with unfolded iterative phase reconstruction. Interspeech, 2018.","title":"End-to-end speech separation with unfolded iterative phase reconstruction","authors":"Zhong-Qiu Wang, Jonathan Le Roux, Deliang Wang, John R Hershey","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2019] Zhong-Qiu Wang, Ke Tan, and DeLiang Wang. Deep learning based phase reconstruction for speaker separation: A trigonometric perspective. In ICASSP, pages 71-75, 2019.","title":"Deep learning based phase reconstruction for speaker separation: A trigonometric perspective","authors":"Zhong-Qiu Wang, Ke Tan, Deliang Wang","year":"2019","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wichern et al., 2019] Gordon Wichern, Joe Antognini, Michael Flynn, Licheng Richard Zhu, Emmett McQuinn, Dwight Crow, Ethan Manilow, and Jonathan Le Roux. Wham!: Extending speech separation to noisy environ- ments. Interspeech, pages 1368-1372, 2019.","title":"Wham!: Extending speech separation to noisy environments","authors":"Gordon Wichern, Joe Antognini, Michael Flynn, Licheng Richard Zhu, Emmett Mcquinn, Dwight Crow, Ethan Manilow, Jonathan Le Roux","year":"2019","venue":"Interspeech","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2020] Liwen Zhang, Ziqiang Shi, Jiqing Han, Anyan Shi, and Ding Ma. Furcanext: End-to-end monau- ral speech separation with dynamic gated dilated temporal convolutional networks. In MMM, pages 653-665, 2020.","title":"Furcanext: End-to-end monaural speech separation with dynamic gated dilated temporal convolutional networks","authors":"Liwen Zhang, Ziqiang Shi, Jiqing Han, Anyan Shi, Ding Ma","year":"2020","venue":"MMM","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}