{"type":"ScholarlyArticle","title":"A De Novo Divide-and-Merge Paradigm for Acoustic Model Optimization in Automatic Speech Recognition","authors":[{"name":"Conghui Tan","slug":"conghui-tan","given":"Conghui","family":"Tan","affiliation":"WeBank Co., Ltd. , Shenzhen , China"},{"name":"Di Jiang","slug":"di-jiang","given":"Di","family":"Jiang","affiliation":"WeBank Co., Ltd. , Shenzhen , China"},{"name":"Jinhua Peng","slug":"jinhua-peng","given":"Jinhua","family":"Peng","affiliation":"WeBank Co., Ltd. , Shenzhen , China"},{"name":"Xueyang Wu","slug":"xueyang-wu","given":"Xueyang","family":"Wu","affiliation":"The Hong Kong University of Science and Technology , Hong Kong"},{"name":"Qian Xu","slug":"qian-xu","given":"Qian","family":"Xu","affiliation":"WeBank Co., Ltd. , Shenzhen , China"},{"name":"Qiang Yang","slug":"qiang-yang","given":"Qiang","family":"Yang","affiliation":"WeBank Co., Ltd. , Shenzhen , China; The Hong Kong University of Science and Technology , Hong Kong"}],"abstract":"Due to the rising awareness of privacy protection and the voluminous scale of speech data, it is becoming infeasible for Automatic Speech Recognition (ASR) system developers to train the acoustic model with complete data as before. In this paper, we propose a novel Divide-and-Merge paradigm to solve salient problems plaguing the ASR field. In the Divide phase, multiple acoustic models are trained based upon different subsets of the complete speech data, while in the Merge phase two novel algorithms are utilized to generate a high-quality acoustic model based upon those trained on data subsets. We first propose the Genetic Merge Algorithm (GMA), which is a highly specialized algorithm for optimizing acoustic models but suffers from low efficiency. We further propose the SGD-Based Optimizational Merge Algorithm (SOMA), which effectively alleviates the efficiency bottleneck of GMA and maintains superior performance. Extensive experiments on public data show that the proposed methods can significantly outperform the state-of-the-art.","doi":"10.24963/ijcai.2020/513","url":"https://ijcai.org/proceedings/2020/513","pdf":"https://ijcai.org/proceedings/2020/0513.pdf","markdown":"https://ijcai.org/proceedings/2020/513.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/513","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Natural Language Processing","track":"Main track","number":513,"pages":{"first":3709,"last":3715},"keywords":["Natural Language Processing: Speech"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24895/file_24895.m3u8","length_sec":262,"provider":"hls"}],"references":[{"position":1,"raw":"[Bahl et al., 1986] Lalit R Bahl, Peter F Brown, Peter V De Souza, and Robert L Mercer. Maximum mutual in- formation estimation of hidden markov model parameters for speech recognition. In proc. icassp, volume 86, pages 49-52, 1986.","title":"Maximum mutual information estimation of hidden markov model parameters for speech recognition","authors":"Peter F Lalit R Bahl, Peter V De Brown, Robert L Souza, Mercer","year":"1986","venue":"proc. icassp","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio et al., 2003] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilis- tic language model. Journal of machine learning research, 3(Feb):1137-1155, 2003.","title":"A neural probabilistic language model","authors":"Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Jauvin","year":"2003","venue":"Journal of machine learning research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen and Ye, 2011] Yunmei Chen and Xiaojing Ye. Pro- jection onto a simplex. arXiv preprint arXiv:1101.6081, 2011.","title":"Projection onto a simplex","authors":"Chen, Ye ;, Yunmei Chen, Xiaojing Ye","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Condat, 2016] Laurent Condat. Fast projection onto the simplex and the simplex and the ℓ 1 ball. Mathematical Programming, 158(1-2):575-585, 2016.","title":"Fast projection onto the simplex and the simplex and the ℓ 1 ball","authors":"Laurent Condat","year":"2016","venue":"Mathematical Programming","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cui and Picheny, 2019] Xiaodong Cui and Michael Picheny. Acoustic model optimization based on evolution- ary stochastic gradient descent with anchors for automatic speech recognition. arXiv preprint arXiv:1907.04882, 2019.","title":"Acoustic model optimization based on evolutionary stochastic gradient descent with anchors for automatic speech recognition","authors":"Xiaodong Cui, Michael Picheny","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Graves et al., 2006] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages 369-376. ACM, 2006.","title":"Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks","authors":"Alex Graves, Santiago Fernández, Faustino Gomez, Jürgen Schmidhuber","year":"2006","venue":"Proceedings of the 23rd international conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hinton et al., 2012] Geoffrey Hinton, Li Deng, Dong Yu, George Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Brian Kingsbury, et al. Deep neural networks for acoustic modeling in speech recognition. IEEE Signal processing magazine, 29, 2012.","title":"Deep neural networks for acoustic modeling in speech recognition","authors":"Geoffrey Hinton, Li Deng, Dong Yu, George Dahl, Abdel-Rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Brian Kingsbury","year":"2012","venue":"IEEE Signal processing magazine","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Holland and others, 1992] John Henry Holland et al. Adap- tation in natural and artificial systems: an introductory analysis with applications to biology, control, and artifi- cial intelligence. MIT press, 1992.","title":"Adaptation in natural and artificial systems: an introductory analysis with applications to biology, control, and artificial intelligence","authors":"John Henry, Holland","year":"1992","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jiang et al., 2019] Di Jiang, Yuanfeng Song, Yongxin Tong, Xueyang Wu, Weiwei Zhao, Qian Xu, and Qiang Yang. Federated topic modeling. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pages 1071-1080, 2019.","title":"Federated topic modeling","authors":"Di Jiang, Yuanfeng Song, Yongxin Tong, Xueyang Wu, Weiwei Zhao, Qian Xu, Qiang Yang","year":"2019","venue":"Proceedings of the 28th ACM International Conference on Information and Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kumar and Gong, 2019] Kshitiz Kumar and Yifan Gong. Static and dynamic state predictions for acoustic model combination. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2782-2786. IEEE, 2019.","title":"Static and dynamic state predictions for acoustic model combination","authors":"Kshitiz Kumar, Yifan Gong","year":"2019","venue":"ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[McMahan et al., 2016] H Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, et al. Communication- efficient learning of deep networks from decentralized data. arXiv preprint arXiv:1602.05629, 2016.","title":"Communicationefficient learning of deep networks from decentralized data","authors":"Eider H Brendan Mcmahan, Daniel Moore, Seth Ramage, Hampson","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Meinedo and Neto, 2000] Hugo Meinedo and Joao P Neto. Combination of acoustic models in continuous speech recognition hybrid systems. In Sixth International Con- ference on Spoken Language Processing, 2000.","title":"Combination of acoustic models in continuous speech recognition hybrid systems","authors":"Neto Meinedo, Hugo Meinedo, Joao P Neto","year":"2000","venue":"Sixth International Conference on Spoken Language Processing","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mikolov et al., 2010] Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černockỳ, and Sanjeev Khudanpur. Re- current neural network based language model. In Eleventh annual conference of the international speech communi- cation association, 2010.","title":"Recurrent neural network based language model","authors":"Mikolov","year":"2010","venue":"Eleventh annual conference of the international speech communication association","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Morin and Bengio, 2005] Frederic Morin and Yoshua Ben- gio. Hierarchical probabilistic neural network language model. In Aistats, volume 5, pages 246-252.","title":"Hierarchical probabilistic neural network language model","authors":"Bengio Morin","year":"2005","venue":"Aistats","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Povey et al., 2011] Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, et al. The kaldi speech recognition toolkit. In IEEE 2011 workshop on automatic speech recognition and understanding, number CONF. IEEE Signal Process- ing Society, 2011.","title":"The kaldi speech recognition toolkit","authors":"Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz","year":"2011","venue":"IEEE 2011 workshop on automatic speech recognition and understanding, number CONF. IEEE Signal Processing Society","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Povey et al., 2014] Daniel Povey, Xiaohui Zhang, and San- jeev Khudanpur. Parallel training of deep neural networks with natural gradient and parameter averaging. arXiv preprint arXiv:1410.7455, 2014.","title":"Parallel training of deep neural networks with natural gradient and parameter averaging","authors":"Daniel Povey, Xiaohui Zhang, Sanjeev Khudanpur","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Stolcke, 2002] Andreas Stolcke. Srilm-an extensible lan- guage modeling toolkit. In Seventh international confer- ence on spoken language processing, 2002.","title":"Srilm-an extensible language modeling toolkit","authors":"Andreas Stolcke","year":"2002","venue":"Seventh international conference on spoken language processing","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Waibel et al., 1989] Alex Waibel, Toshiyuki Hanazawa, Ge- offrey Hinton, Kiyohiro Shikano, and Kevin J Lang. Phoneme recognition using time-delay neural networks. IEEE transactions on acoustics, speech, and signal pro- cessing, 37(3):328-339, 1989.","title":"Phoneme recognition using time-delay neural networks","authors":"Alex Waibel, Toshiyuki Hanazawa, Geoffrey Hinton, Kiyohiro Shikano, Kevin J Lang","year":"1989","venue":"IEEE transactions on acoustics, speech, and signal processing","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Xiong et al., 2018] Wayne Xiong, Lingfeng Wu, Fil All- eva, Jasha Droppo, Xuedong Huang, and Andreas Stol- cke. The microsoft 2017 conversational speech recogni- tion system. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5934-5938. IEEE, 2018.","title":"The microsoft 2017 conversational speech recognition system","authors":"Wayne Xiong, Lingfeng Wu, Fil Alleva, Jasha Droppo, Xuedong Huang, Andreas Stolcke","year":"2018","venue":"IEEE international conference on acoustics, speech and signal processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhang et al., 2013] Shanshan Zhang, Ce Zhang, Zhao You, Rong Zheng, and Bo Xu. Asynchronous stochastic gradi- ent descent for dnn training. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 6660-6663. IEEE, 2013.","title":"Asynchronous stochastic gradient descent for dnn training","authors":"Shanshan Zhang, Ce Zhang, Zhao You, Rong Zheng, Bo Xu","year":"2013","venue":"2013 IEEE International Conference on Acoustics, Speech and Signal Processing","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhou, 2012] Zhi-Hua Zhou. Ensemble methods: founda- tions and algorithms. Chapman and Hall/CRC, 2012.","title":"Ensemble methods: foundations and algorithms","authors":"Zhi-Hua Zhou","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}