{"type":"ScholarlyArticle","title":"Exploration of Tree-based Hierarchical Softmax for Recurrent Language Models","authors":[{"name":"Nan Jiang","slug":"nan-jiang","given":"Nan","family":"Jiang","affiliation":"† State Key Laboratory of Software Development Environment , Beihang University , China; ‡ School of Computer Science and Engineering , Beihang University , China"},{"name":"Wenge Rong","slug":"wenge-rong","given":"Wenge","family":"Rong","affiliation":"† State Key Laboratory of Software Development Environment , Beihang University , China; ‡ School of Computer Science and Engineering , Beihang University , China"},{"name":"Min Gao","slug":"min-gao","given":"Min","family":"Gao","affiliation":"§ School of Software Engineering , Chongqing University , China"},{"name":"Yikang Shen","slug":"yikang-shen","given":"Yikang","family":"Shen","affiliation":"Montréal Institute for Learning Algorithms , Universté de Montréal , Canada"},{"name":"Zhang Xiong","slug":"zhang-xiong","given":"Zhang","family":"Xiong","affiliation":"† State Key Laboratory of Software Development Environment , Beihang University , China; ‡ School of Computer Science and Engineering , Beihang University , China"}],"abstract":"Recently, variants of neural networks for computational linguistics have been proposed and successfully applied to neural language modeling and neural machine translation. These neural models can leverage knowledge from massive corpora but they are extremely slow as they predict candidate words from a large vocabulary during training and inference. As an alternative to gradient approximation and softmax with class decomposition, we explore the tree-based hierarchical softmax method and reform its architecture, making it compatible with modern GPUs and introducing a compact tree-based loss function. When combined with several word hierarchical clustering algorithms, improved performance is achieved in language modelling task with intrinsic evaluation criterions on PTB, WikiText-2 and WikiText-103 datasets.","doi":"10.24963/ijcai.2017/271","url":"https://ijcai.org/proceedings/2017/271","pdf":"https://ijcai.org/proceedings/2017/0271.pdf","markdown":"https://ijcai.org/proceedings/2017/271.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/271","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":271,"pages":{"first":1951,"last":1957},"keywords":["Natural Language Processing: Natural Language Generation","Natural Language Processing: Natural Language Processing","Machine Learning: Machine Learning","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Baltescu and Blunsom, 2015] Paul Baltescu and Phil Blun- som. Pragmatic neural language modelling in machine translation. In Proc. of NAACL-HLT, pages 820-829, 2015.","title":"Paul Baltescu and Phil Blunsom. Pragmatic neural language modelling in machine translation","authors":"Blunsom Baltescu","year":"2015","venue":"Proc. of NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio et al., 2003] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. A neural proba- bilistic language model. Journal of Machine Learning Re- search, 3:1137-1155, 2003.","title":"A neural probabilistic language model","authors":"Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Janvin","year":"2003","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brown et al., 1992] Peter F. Brown, Vincent J. Della Pietra, Peter V. de Souza, Jennifer C. Lai, and Robert L. Mercer. Class-based n-gram models of natural language. Compu- tational Linguistics, 18(4):467-479, 1992.","title":"Class-based n-gram models of natural language","authors":"F Peter, Brown, J Vincent, Peter V Della Pietra, Jennifer C De Souza, Robert L Lai, Mercer","year":"1992","venue":"Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2016] Wenlin Chen, David Grangier, and Michael Auli. Strategies for training large vocabulary neu- ral language models. In Proc. of ACL, pages 1975-1985, 2016.","title":"Strategies for training large vocabulary neural language models","authors":"Chen","year":"2016","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chung et al., 2014] Junyoung Chung, C ¸aglar Gülc ¸ehre, KyungHyun Cho, and Yoshua Bengio. Empirical evalua- tion of gated recurrent neural networks on sequence mod- eling. CoRR, abs/1412.3555, 2014.","title":"Empirical evaluation of gated recurrent neural networks on sequence modeling","authors":"Chung","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dugas et al., 2000] Charles Dugas, Yoshua Bengio, Franc ¸ois Bélisle, Claude Nadeau, and René Garcia. Incorporating second-order functional knowledge for better option pricing. In Proc. of NIPS, pages 472-478, 2000.","title":"Incorporating second-order functional knowledge for better option pricing","authors":"Charles Dugas, Yoshua Bengio, Claude Franc ¸ois Bélisle, René Nadeau, Garcia","year":"2000","venue":"Proc. of NIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Goodman, 2001] Joshua Goodman. Classes for fast maxi- mum entropy training. In Proc. of ICASSP, pages 561- 564, 2001.","title":"Classes for fast maximum entropy training","authors":"Joshua Goodman, Goodman","year":"2001","venue":"Proc. of ICASSP","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gutmann and Hyvärinen, 2010] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proc. of AISTATS, pages 297-304, 2010.","title":"Noise-contrastive estimation: A new estimation principle for unnormalized statistical models","authors":"Hyvärinen Gutmann, Michael Gutmann, Aapo Hyvärinen","year":"2010","venue":"Proc. of AISTATS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gutmann and Hyvärinen, 2012] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation of unnor- malized statistical models, with applications to natural image statistics. Journal of Machine Learning Research, 13:307-361, 2012.","title":"Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics","authors":"Hyvärinen Gutmann, Michael Gutmann, Aapo Hyvärinen","year":"2012","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hochreiter, 1998] Sepp Hochreiter. The vanishing gradient problem during learning recurrent neural nets and problem solutions. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 6(2):107-116, 1998.","title":"The vanishing gradient problem during learning recurrent neural nets and problem solutions","authors":"Sepp Hochreiter","year":"1998","venue":"International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jean et al., 2015] Sébastien Jean, KyungHyun Cho, Roland Memisevic, and Yoshua Bengio. On using very large tar- get vocabulary for neural machine translation. In Proc. of ACL, pages 1-10, 2015.","title":"On using very large target vocabulary for neural machine translation","authors":"Sébastien Jean, Kyunghyun Cho, Roland Memisevic, Yoshua Bengio","year":"2015","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ji et al., 2016] Shihao Ji, S. V. N. Vishwanathan, Nadathur Satish, Michael J. Anderson, and Pradeep Dubey. Black- out: Speeding up recurrent neural network language mod- els with very large vocabularies. In Proc. of ICLR, 2016.","title":"Blackout: Speeding up recurrent neural network language models with very large vocabularies","authors":"Ji Shihao, S V N Vishwanathan, Nadathur Satish, Michael J Anderson, Pradeep Dubey","year":"2016","venue":"Proc. of ICLR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kim et al., 2016] Yoon Kim, Yacine Jernite, David Sontag, and Alexander M. Rush. Character-aware neural language models. In Proc. of AAAI, pages 2741-2749, 2016.","title":"Character-aware neural language models","authors":"Yoon Kim, Yacine Jernite, David Sontag, Alexander M Rush","year":"2016","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kleinbaum and Klein, 2010] David G Kleinbaum and Mitchel Klein. Maximum likelihood techniques: An overview. Logistic regression, pages 103-127, 2010.","title":"Maximum likelihood techniques: An overview","authors":"Klein ; Kleinbaum, G David, Mitchel Kleinbaum, Klein","year":"2010","venue":"Logistic regression","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liang, 2005] Percy Liang. Semi-supervised learning for natural language. PhD thesis, MIT, 2005.","title":"Semi-supervised learning for natural language","authors":"Percy Liang","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Marcus et al., 1993] Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotat- ed corpus of english: The penn treebank. Computational Linguistics, 19(2):313-330, 1993.","title":"Building a large annotated corpus of english: The penn treebank","authors":"Mitchell P Marcus, Beatrice Santorini, Mary Ann Marcinkiewicz","year":"1993","venue":"Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Merity et al., 2017] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In Proc. of ICLR, 2017.","title":"Pointer sentinel mixture models","authors":"Stephen Merity, Caiming Xiong, James Bradbury, Richard Socher","year":"2017","venue":"Proc. of ICLR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mikolov et al., 2010] Tomas Mikolov, Martin Karafiát, Lukás Burget, Jan Cernocký, and Sanjeev Khudanpur. Re- current neural network based language model. In Proc. of Interspeech, pages 1045-1048, 2010.","title":"Recurrent neural network based language model","authors":"Mikolov","year":"2010","venue":"Proc. of Interspeech","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mikolov et al., 2013a] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. CoRR, abs/1301.3781, 2013.","title":"Efficient estimation of word representations in vector space","authors":"Mikolov","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mikolov et al., 2013b] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositionality. In Proc. of NIPS, pages 3111-3119. 2013.","title":"Distributed representations of words and phrases and their compositionality","authors":"Mikolov","year":"2013","venue":"Proc. of NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Mnih and Hinton, 2009] Andriy Mnih and Geoffrey E Hin- ton. A scalable hierarchical distributed language model. In Proc. of NIPS, pages 1081-1088. 2009.","title":"A scalable hierarchical distributed language model","authors":"Andriy Mnih, Geoffrey E Hinton","year":"2009","venue":"Proc. of NIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Mnih and Kavukcuoglu, 2013] Andriy Mnih and Koray Kavukcuoglu. Learning word embeddings efficiently with noise-contrastive estimation. In Proc. of NIPS, pages 2265-2273. 2013.","title":"Learning word embeddings efficiently with noise-contrastive estimation","authors":"Kavukcuoglu Mnih, Andriy Mnih, Koray Kavukcuoglu","year":"2013","venue":"Proc. of NIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Mnih and Teh, 2012] Andriy Mnih and Yee Whye Teh. A fast and simple algorithm for training neural probabilistic language models. In Proc. of ICML, 2012.","title":"A fast and simple algorithm for training neural probabilistic language models","authors":"Andriy Mnih, Yee Whye Teh","year":"2012","venue":"Proc. of ICML","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Morin and Bengio, 2005] Frederic Morin and Yoshua Ben- gio. Hierarchical probabilistic neural network language model. In Proc. of AISTATS, 2005.","title":"Hierarchical probabilistic neural network language model","authors":"Bengio Morin","year":"2005","venue":"Proc. of AISTATS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Schwenk, 2007] Holger Schwenk. Continuous space lan- guage models. Computer Speech & Language, 21(3):492- 518, 2007.","title":"Continuous space language models","authors":"Holger Schwenk","year":"2007","venue":"Computer Speech & Language","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sennrich et al., 2016] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proc. of ACL, 2016.","title":"Neural machine translation of rare words with subword units","authors":"Sennrich","year":"2016","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Theano Development Team, 2016] Theano Development Team. Theano: A Python framework for fast computation of mathematical expressions. CoRR, abs/1605.02688, 2016.","title":"Theano Development Team. Theano: A Python framework for fast computation of mathematical expressions","authors":null,"year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang and Wang, 2016] ZhongQiu Wang and DeLiang Wang. A joint training framework for robust automatic speech recognition. IEEE/ACM Transactions on Audio, Speech & Language Processing, 24(4):796-806, 2016.","title":"A joint training framework for robust automatic speech recognition","authors":"Wang, Wang ;, Zhongqiu Wang, Deliang Wang","year":"2016","venue":"IEEE/ACM Transactions on Audio, Speech & Language Processing","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}