{"type":"ScholarlyArticle","title":"Fast Parallel Training of Neural Language Models","authors":[{"name":"Tong Xiao","slug":"tong-xiao","given":"Tong","family":"Xiao","affiliation":"† NiuTrans Lab. , Northeastern University , Shenyang 110819 , China"},{"name":"Jingbo Zhu","slug":"jingbo-zhu","given":"Jingbo","family":"Zhu","affiliation":"† NiuTrans Lab. , Northeastern University , Shenyang 110819 , China"},{"name":"Tongran Liu","slug":"tongran-liu","given":"Tongran","family":"Liu","affiliation":"‡ Institute of Psychology (CAS) , Beijing 100101 , China"},{"name":"Chunliang Zhang","slug":"chunliang-zhang","given":"Chunliang","family":"Zhang","affiliation":"† NiuTrans Lab. , Northeastern University , Shenyang 110819 , China"}],"abstract":"Training neural language models (NLMs) is very time consuming and we need parallelization for system speedup. However, standard training methods have poor scalability across multiple devices (e.g., GPUs) due to the huge time cost required to transmit data for gradient sharing in the back-propagation process. In this paper we present a sampling-based approach to reducing data transmission for better scaling of NLMs. As a ''bonus'', the resulting model also improves the training speed on a single device. Our approach yields significant speed improvements on a recurrent neural network-based language model. On four NVIDIA GTX1080 GPUs, it achieves a speedup of 2.1+ times over the standard asynchronous stochastic gradient descent baseline, yet with no increase in perplexity. This is even 4.2 times faster than the naive single GPU counterpart.","doi":"10.24963/ijcai.2017/586","url":"https://ijcai.org/proceedings/2017/586","pdf":"https://ijcai.org/proceedings/2017/0586.pdf","markdown":"https://ijcai.org/proceedings/2017/586.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/586","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Natural Language Processing","track":"Main track","number":586,"pages":{"first":4193,"last":4199},"keywords":["Natural Language Processing: NLP Applications and Tools","Natural Language Processing: Natural Language Processing"],"videos":[],"references":[{"position":1,"raw":"[Agarwal and Duchi, 2011] Alekh Agarwal and John C Duchi. Distributed delayed stochastic optimization. In Proceedings of the 25th Annual Conference on Neural Information Processing Systems (NIPS), pages 873-881, Granada, Spain, 2011.","title":"Distributed delayed stochastic optimization","authors":"Alekh Agarwal, John C Duchi","year":"2011","venue":"Proceedings of the 25th Annual Conference on Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio and Senécal, 2003] Yoshua Bengio and Jean- Sbastien Senécal. Quick training of probabilistic neural nets by importance sampling. In Proceedings of AISTATS, 2003.","title":"Quick training of probabilistic neural nets by importance sampling","authors":"Senécal Bengio, Yoshua Bengio, Jean-Sbastien Senécal","year":"2003","venue":"Proceedings of AISTATS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bengio and Senécal, 2008] Yoshua Bengio and Jean- Sébastien Senécal. Adaptive importance sampling to accelerate training of a neural probabilistic language mod- el. IEEE Transactions on Neural Network, 4:713-722, 2008.","title":"Adaptive importance sampling to accelerate training of a neural probabilistic language model","authors":"Senécal Bengio, Yoshua Bengio, Jean-Sébastien Senécal","year":"2008","venue":"IEEE Transactions on Neural Network","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bengio et al., 2003] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural proba- bilistic language model. Journal of Machine Learning Re- search, 3:1137-1155, 2003.","title":"A neural probabilistic language model","authors":"Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Jauvin","year":"2003","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dean et al., 2012] Jeffrey Dean, Greg Corrado, Rajat Mon- ga, Kai Chen, Matthieu Devin, Q Le, Mark Z Mao, Mar- caurelio Ranzato, Andrew W Senior, Paul Tucker, et al. Large scale distributed deep networks. Information pro- cessing systems, pages 1223-1231, 2012.","title":"Large scale distributed deep networks","authors":"Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Q Le, Mark Z Mao, Marcaurelio Ranzato, Andrew W Senior, Paul Tucker","year":"2012","venue":"Information processing systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hoaglin et al., 1983] David C. Hoaglin, Frederick Mosteller, and John W. Tukey. Understanding Ro- bust and Exploratory Data Analysis. John Wiley, 1983.","title":"Understanding Robust and Exploratory Data Analysis","authors":"David C Hoaglin, Frederick Mosteller, John W Tukey","year":"1983","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735-17808, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Józefowicz et al., 2016] Rafal Józefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Ex- ploring the limits of language modeling. CoRR, ab- s/1602.02410, 2016.","title":"Exploring the limits of language modeling","authors":"Rafal Józefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, Yonghui Wu","year":"2016","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Langford et al., 2009] John Langford, Alexander J Smola, and Martin Zinkevich. Slow learners are fast. In Proceed- ings of the 23rd Annual Conference on Neural Information Processing Systems (NIPS), pages 2331-2339, Vancouver, Canada, 2009.","title":"Slow learners are fast","authors":"John Langford, Alexander J Smola, Martin Zinkevich","year":"2009","venue":"Proceedings of the 23rd Annual Conference on Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Mann et al., 2009] Gideon Mann, Ryan T. McDonald, Mehryar Mohri, and Dan Walker. Efficient large-scale dis- tributed training of conditional maximum entropy models. In Proceedings of the 23rd Annual Conference on Neu- ral Information Processing Systems (NIPS), pages 1231- 1239, Vancouver, Canada, 2009.","title":"Efficient large-scale distributed training of conditional maximum entropy models","authors":"Gideon Mann, Ryan T Mcdonald, Mehryar Mohri, Dan Walker","year":"2009","venue":"Proceedings of the 23rd Annual Conference on Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[McDonald et al., 2010] Ryan McDonald, Keith Hall, and Gideon Mann. Distributed training strategies for the struc- tured perceptron. In Proceedings of Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 456-464, Los Angeles, California, June 2010.","title":"Distributed training strategies for the structured perceptron","authors":"Ryan Mcdonald, Keith Hall, Gideon Mann","year":"2010","venue":"Proceedings of Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mikolov et al., 2010] Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Černocký Jan, and Sanjeev Khudanpur. Re- current neural network based language model. In Pro- ceedings of INTERSPEECH, pages 1045-1048, Makuhari, Chiba, Japan, September 2010.","title":"Recurrent neural network based language model","authors":"Mikolov","year":"2010","venue":"Proceedings of INTERSPEECH","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mnih and Hinton, 2008] Andriy Mnih and Geoffrey Hinton. A scalable hierarchical distributed language model. In Proceedings of NIPS, pages 1081-1088, Vancouver, Cana- da, December 2008.","title":"A scalable hierarchical distributed language model","authors":"Andriy Mnih, Geoffrey Hinton","year":"2008","venue":"Proceedings of NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mnih and Teh, 2012] Andriy Mnih and Yee Whye Teh. A fast and simple algorithm for training neural probabilistic language models. In Processings of the 29th Internation- al Conference on Machine Learning (ICML), pages 1751- 1758, Edinburgh, Scotland, June 2012.","title":"A fast and simple algorithm for training neural probabilistic language models","authors":"Andriy Mnih, Yee Whye Teh","year":"2012","venue":"Processings of the 29th International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Morin and Bengio, 2005] Frederic Morin and Yoshua Ben- gio. Hierarchical probabilistic neural network language model. In Proceedings of AISTATS, pages 246-252, 2005.","title":"Hierarchical probabilistic neural network language model","authors":"Bengio Morin","year":"2005","venue":"Proceedings of AISTATS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Niu et al., 2011] Feng Niu, Benjaminm Recht, Christopher Ré, and Stephen J. Wright. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In Proceed- ings of the 25th Annual Conference on Neural Information Processing Systems (NIPS), Granada, Spain, 2011.","title":"Hogwild: A lock-free approach to parallelizing stochastic gradient descent","authors":"Feng Niu, Benjaminm Recht, Christopher Ré, Stephen J Wright","year":"2011","venue":"Proceedings of the 25th Annual Conference on Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Seide et al., 2014] Frank Seide, Hao Fu, Jasha Droppo, Gang Li, and Dong Yu. 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech dnns. In Proceedings of INTERSPEECH, pages 1058-1062, Singapore, September 2014.","title":"1-bit stochastic gradient descent and its application to data-parallel distributed training of speech dnns","authors":"Frank Seide, Hao Fu, Jasha Droppo, Gang Li, Dong Yu","year":"2014","venue":"Proceedings of INTERSPEECH","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Srivastava et al., 2014] Nitish Srivastava, Geoffrey E Hin- ton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929-1958, 2014.","title":"Dropout: a simple way to prevent neural networks from overfitting","authors":"Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov","year":"2014","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Xiao et al., 2012] Tong Xiao, Jingbo Zhu, Hao Zhang, and Qiang Li. NiuTrans: An Open Source Toolkit for Phrase- based and Syntax-based Machine Translation. In Proceed- ings of the 50th Annual Meeting of the Association for Computational Linguistic (ACL): System Demonstrations, pages 19-24, Jeju Island, Korea, July 2012.","title":"NiuTrans: An Open Source Toolkit for Phrasebased and Syntax-based Machine Translation","authors":"Tong Xiao, Jingbo Zhu, Hao Zhang, Qiang Li","year":"2012","venue":"Proceedings of the 50th Annual Meeting of the Association for Computational Linguistic (ACL): System Demonstrations","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Zhang et al., 2015] Sixin Zhang, Anna Choromanska, and Yann Lecun. Deep learning with elastic averaging sgd. In Proceedings of the Twenty-ninth Annual Conference on Neural Information Processing Systems (NIPS), pages 685-693, Montréal, Canada, 2015.","title":"Deep learning with elastic averaging sgd","authors":"Zhang","year":"2015","venue":"Proceedings of the Twenty-ninth Annual Conference on Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zinkevich et al., 2010] Martin A. Zinkevich, Alex Smola, Markus Weimer, and Lihong Li. Parallelized stochastic gradient descent. In Proceedings of the 24th Annual Con- ference on Neural Information Processing Sytems (NIPS), pages 2595-2603, Vancouver, Canada, 2010.","title":"Parallelized stochastic gradient descent","authors":"Martin A Zinkevich, Alex Smola, Markus Weimer, Lihong Li","year":"2010","venue":"Proceedings of the 24th Annual Conference on Neural Information Processing Sytems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zoph et al., 2016] Barret Zoph, Ashish Vaswani, Jonathan May, and Kevin Knight. Simple, fast noise-contrastive es- timation for large rnn vocabularies. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, pages 1217-1222, San Diego, Cali- fornia, June 2016.","title":"Simple, fast noise-contrastive estimation for large rnn vocabularies","authors":"Barret Zoph, Ashish Vaswani, Jonathan May, Kevin Knight","year":"2016","venue":"Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}