{"type":"ScholarlyArticle","title":"Listen, Think and Listen Again: Capturing Top-down Auditory Attention for Speaker-independent Speech Separation","authors":[{"name":"Jing Shi","slug":"jing-shi","given":"Jing","family":"Shi","affiliation":"Institute of Automation , Chinese Academy of Sciences (CASIA) , Beijing , China; Research Center for Brain-inspired Intelligence , CASIA; University of Chinese Academy of Sciences"},{"name":"Jiaming Xu","slug":"jiaming-xu","given":"Jiaming","family":"Xu","affiliation":"Institute of Automation , Chinese Academy of Sciences (CASIA) , Beijing , China; Research Center for Brain-inspired Intelligence , CASIA"},{"name":"Guangcan Liu","slug":"guangcan-liu","given":"Guangcan","family":"Liu","affiliation":"Institute of Automation , Chinese Academy of Sciences (CASIA) , Beijing , China; Research Center for Brain-inspired Intelligence , CASIA; University of Chinese Academy of Sciences"},{"name":"Bo Xu","slug":"bo-xu","given":"Bo","family":"Xu","affiliation":"Institute of Automation , Chinese Academy of Sciences (CASIA) , Beijing , China; Research Center for Brain-inspired Intelligence , CASIA; University of Chinese Academy of Sciences; Center for Excellence in Brain Science and Intelligence Technology , CAS. China"}],"abstract":"Recent deep learning methods have made significant progress in multi-talker mixed speech separation. However, most existing models adopt a driftless strategy to separate all the speech channels rather than selectively attend the target one. As a result, those frameworks may be failed to offer a satisfactory solution in complex auditory scene where the number of input sounds is usually uncertain and even dynamic. In this paper, we present a novel neural network based structure motivated by the top-down attention behavior of human when facing complicated acoustical scene. Different from previous works, our method constructs an inference-attention structure to predict interested candidates and extract each speech channel of them. Our work gets rid of the limitation that the number of channels must be given or the high computation complexity for label permutation problem. We evaluated our model on the WSJ0 mixed-speech tasks. In all the experiments, our model gets highly competitive to reach and even outperform the baselines.","doi":"10.24963/ijcai.2018/605","url":"https://ijcai.org/proceedings/2018/605","pdf":"https://ijcai.org/proceedings/2018/0605.pdf","markdown":"https://ijcai.org/proceedings/2018/605.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/605","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Natural Language Processing","track":"Main track","number":605,"pages":{"first":4353,"last":4360},"keywords":["Natural Language Processing: Speech","Machine Learning: Neural Networks","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Arik et al., 2017] Sercan Arik, Gregory Diamos, Andrew Gibiansky, John Miller, Kainan Peng, Wei Ping, Jonathan Raiman, and Yanqi Zhou. Deep voice 2: Multi-speaker neural text-to-speech. arXiv preprint arXiv:1705.08947, 2017.","title":"Deep voice 2: Multi-speaker neural text-to-speech","authors":"Sercan Arik, Gregory Diamos, Andrew Gibiansky, John Miller, Kainan Peng, Wei Ping, Jonathan Raiman, Yanqi Zhou","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Beck and Kastner, 2009] Diane M Beck and Sabine Kast- ner. Top-down and bottom-up mechanisms in biasing com- petition in the human brain. Vision research, 49(10):1154- 1165, 2009.","title":"Top-down and bottom-up mechanisms in biasing competition in the human brain","authors":"M Diane, Sabine Beck, Kastner","year":"2009","venue":"Vision research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bregman, 1990] AS Bregman. Auditory scene analysis: The perceptual organization of sound. 1990, 1990.","title":"Auditory scene analysis: The perceptual organization of sound","authors":"; As Bregman, Bregman","year":"1990","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brown and Cooke, 1994] Guy J Brown and Martin Cooke. Computational auditory scene analysis. Computer Speech & Language, 8(4):297-336, 1994.","title":"Computational auditory scene analysis","authors":"J Guy, Martin Brown, Cooke","year":"1994","venue":"Computer Speech & Language","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cao et al., 2015] Chunshui Cao, Xianming Liu, Yi Yang, Yinan Yu, Jiang Wang, Zilei Wang, Yongzhen Huang, Liang Wang, Chang Huang, Wei Xu, et al. Look and think twice: Capturing top-down visual attention with feedback convolutional neural networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 2956-2964, 2015.","title":"Look and think twice: Capturing top-down visual attention with feedback convolutional neural networks","authors":"Cao","year":"2015","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2017] Zhuo Chen, Yi Luo, and Nima Mes- garani. Deep attractor network for single-microphone speaker separation. In Acoustics, Speech and Signal Pro- cessing (ICASSP), 2017 IEEE International Conference on, pages 246-250. IEEE, 2017.","title":"Deep attractor network for single-microphone speaker separation","authors":"Zhuo Chen, Yi Luo, Nima Mesgarani","year":"2017","venue":"Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen, 2017] Zhuo Chen. Single Channel auditory source separation with neural network. PhD thesis, Columbia University, 2017.","title":"Single Channel auditory source separation with neural network","authors":"Zhuo Chen","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cherry, 1953] E Colin Cherry. Some experiments on the recognition of speech, with one and with two ears. Jour- nal of the Acoustical Society of America, 25(5):975-979, 1953.","title":"Some experiments on the recognition of speech, with one and with two ears","authors":"Colin Cherry","year":"1953","venue":"Journal of the Acoustical Society of America","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ciocca, 2008] Valter Ciocca. The auditory organization of complex sounds. Frontiers in bioscience: a journal and virtual library, 13:148-169, 2008.","title":"The auditory organization of complex sounds","authors":"Valter Ciocca","year":"2008","venue":"Frontiers in bioscience: a journal and virtual library","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Elhilali, 2017] Mounya Elhilali. Modeling the Cocktail Party Problem, pages 111-135. Springer International Publishing, Cham, 2017.","title":"Mounya Elhilali. Modeling the Cocktail Party Problem","authors":"Elhilali","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hershey et al., 2016] John R Hershey, Zhuo Chen, Jonathan Le Roux, and Shinji Watanabe. Deep clustering: Discrim- inative embeddings for segmentation and separation. In Acoustics, Speech and Signal Processing (ICASSP), pages 31-35. IEEE, 2016.","title":"Deep clustering: Discriminative embeddings for segmentation and separation","authors":"Zhuo John R Hershey, Jonathan Le Chen, Shinji Roux, Watanabe","year":"2016","venue":"Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Huang et al., 2014] Po-Sen Huang, Minje Kim, Mark Hasegawa-Johnson, and Paris Smaragdis. Deep learn- ing for monaural speech separation. In Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on, pages 1562-1566. IEEE, 2014.","title":"Deep learning for monaural speech separation","authors":"Huang","year":"2014","venue":"Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Isik et al., 2016] Yusuf Isik, Jonathan Le Roux, Zhuo Chen, Shinji Watanabe, and John R Hershey. Single-channel multi-speaker separation using deep clustering. arXiv preprint arXiv:1607.02173, 2016.","title":"Single-channel multi-speaker separation using deep clustering","authors":"Yusuf Isik, Jonathan Le Roux, Zhuo Chen, Shinji Watanabe, John R Hershey","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kaya and Elhilali, 2017] Emine Merve Kaya and Mounya Elhilali. Modelling auditory attention. Phil. Trans. R. Soc. B, 372(1714):20160101, 2017.","title":"Emine Merve Kaya and Mounya Elhilali","authors":"Elhilali Kaya","year":"1714","venue":"Phil. Trans. R. Soc. B","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2017] Chao Li, Xiaokong Ma, Bing Jiang, Xi- angang Li, Xuewei Zhang, Xiao Liu, Ying Cao, Ajay Kannan, and Zhenyao Zhu. Deep speaker: an end-to- end neural speaker embedding system. arXiv preprint arXiv:1705.02304, 2017.","title":"Deep speaker: an end-toend neural speaker embedding system","authors":"Chao Li, Xiaokong Ma, Bing Jiang, Xiangang Li, Xuewei Zhang, Xiao Liu, Ying Cao, Ajay Kannan, Zhenyao Zhu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Luong et al., 2015] Thang Luong, Hieu Pham, and Christo- pher D Manning. Effective approaches to attention-based neural machine translation. empirical methods in natural language processing, pages 1412-1421, 2015.","title":"Effective approaches to attention-based neural machine translation","authors":"Thang Luong, Hieu Pham, Christopher D Manning","year":"2015","venue":"empirical methods in natural language processing","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[O'sullivan et al., 2014] James A O'sullivan, Alan J Power, Nima Mesgarani, Siddharth Rajaram, John J Foxe, Bar- bara G Shinn-Cunningham, Malcolm Slaney, Shihab A Shamma, and Edmund C Lalor. Attentional selection in a cocktail party environment can be decoded from single- trial eeg. Cerebral Cortex, 25(7):1697-1706, 2014.","title":"Attentional selection in a cocktail party environment can be decoded from singletrial eeg","authors":"A James, Alan J O'sullivan, Nima Power, Siddharth Mesgarani, John J Rajaram, Barbara G Foxe, Malcolm Shinn-Cunningham, Shihab A Slaney, Edmund C Shamma, Lalor","year":"2014","venue":"Cerebral Cortex","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Schmidt and Olsson, 2006] Mikkel N Schmidt and Ras- mus Kongsgaard Olsson. Single-channel speech separa- tion using sparse non-negative matrix factorization. In Spoken Language Proceesing, ISCA International Confer- ence on (INTERSPEECH), 2006.","title":"Single-channel speech separation using sparse non-negative matrix factorization","authors":"Olsson ; Schmidt, N Mikkel, Rasmus Schmidt, Kongsgaard Olsson","year":"2006","venue":"Spoken Language Proceesing, ISCA International Conference on (INTERSPEECH)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Vincent et al., 2006] Emmanuel Vincent, Rémi Gribonval, and Cédric Févotte. Performance measurement in blind audio source separation. IEEE transactions on audio, speech, and language processing, 14(4):1462-1469, 2006.","title":"Performance measurement in blind audio source separation","authors":"Emmanuel Vincent, Rémi Gribonval, Cédric Févotte","year":"2006","venue":"IEEE transactions on audio, speech, and language processing","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2016] Guan Xiang Wang, Chung Chien Hsu, and Jen Tzung Chien. Discriminative deep recurrent neu- ral networks for monaural speech separation. In IEEE In- ternational Conference on Acoustics, Speech and Signal Processing, pages 2544-2548, 2016.","title":"Discriminative deep recurrent neural networks for monaural speech separation","authors":"Guan Xiang, Wang, Chung Chien Hsu, Jen Tzung Chien","year":"2016","venue":"IEEE International Conference on Acoustics, Speech and Signal Processing","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang, 2005] DeLiang Wang. On ideal binary mask as the computational goal of auditory scene analysis. Speech sep- aration by humans and machines, pages 181-197, 2005.","title":"On ideal binary mask as the computational goal of auditory scene analysis","authors":"Wang ;, Deliang Wang","year":"2005","venue":"Speech separation by humans and machines","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xu et al., 2018] Jiaming Xu, Jing Shi, Guangcan Liu, Xi- uyi Chen, and Bo Xu. Modeling attention and memory for auditory selection in a cocktail party environment. In Proceedings of the 32nd AAAI Conference on Artificial In- telligence (AAAI), 2018.","title":"Modeling attention and memory for auditory selection in a cocktail party environment","authors":"Jiaming Xu, Jing Shi, Guangcan Liu, Xiuyi Chen, Bo Xu","year":"2018","venue":"Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yu et al., 2017] Dong Yu, Morten Kolbæk, Zheng-Hua Tan, and Jesper Jensen. Permutation invariant training of deep models for speaker-independent multi-talker speech sep- aration. In Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE International Conference on, pages 241-245. IEEE, 2017.","title":"Permutation invariant training of deep models for speaker-independent multi-talker speech separation","authors":"Dong Yu, Morten Kolbæk, Zheng-Hua Tan, Jesper Jensen","year":"2017","venue":"Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}