{"type":"ScholarlyArticle","title":"Compressed Nonparametric Language Modelling","authors":[{"name":"Ehsan Shareghi","slug":"ehsan-shareghi","given":"Ehsan","family":"Shareghi","affiliation":"Faculty of Information Technology , Monash University ♠ Computing and Information Systems , The University of Melbourne"},{"name":"Gholamreza Haffari","slug":"gholamreza-haffari","given":"Gholamreza","family":"Haffari","affiliation":"Faculty of Information Technology , Monash University ♠ Computing and Information Systems , The University of Melbourne"},{"name":"Trevor Cohn","slug":"trevor-cohn","given":"Trevor","family":"Cohn","affiliation":"Faculty of Information Technology , Monash University ♠ Computing and Information Systems , The University of Melbourne"}],"abstract":"Hierarchical Pitman-Yor Process priors are compelling for learning language models, outperforming point-estimate based methods. However, these models remain unpopular due to computational and statistical inference issues, such as memory and time usage, as well as poor mixing of sampler. In this work we propose a novel framework which represents the HPYP model compactly using compressed suffix trees. Then, we develop an efficient approximate inference scheme in this framework that has a much lower memory footprint compared to full HPYP and is fast in the inference time. The experimental results illustrate that our model can be built on significantly larger datasets compared to previous HPYP models, while being several orders of magnitudes smaller, fast for training and inference, and outperforming the perplexity of the state-of-the-art Modified Kneser-Ney count-based LM smoothing by up to 15%.","doi":"10.24963/ijcai.2017/376","url":"https://ijcai.org/proceedings/2017/376","pdf":"https://ijcai.org/proceedings/2017/0376.pdf","markdown":"https://ijcai.org/proceedings/2017/376.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/376","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning S-Z","track":"Main track","number":376,"pages":{"first":2701,"last":2707},"keywords":["Machine Learning: Machine Learning","Natural Language Processing: Natural Language Processing","Uncertainty in AI: Approximate Probabilistic Inference"],"videos":[],"references":[{"position":1,"raw":"[Buck et al., 2014] Christian Buck, Kenneth Heafield, and Bas van Ooyen. N-gram counts and language models from the common crawl. In Proceedings of the Language Re- sources and Evaluation Conference, 2014.","title":"N-gram counts and language models from the common crawl","authors":"Buck","year":"2014","venue":"Proceedings of the Language Resources and Evaluation Conference","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Buntine and Hutter, 2012] Wray Buntine and Marcus Hut- ter. A bayesian view of the Poisson-Dirichlet process. arXiv preprint arXiv:1007.0296, 2012.","title":"A bayesian view of the Poisson-Dirichlet process","authors":"Buntine, ; Hutter, Marcus Buntine, Hutter","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Burrows and Wheeler, 1994] Michael Burrows and David Wheeler. A block sorting lossless data compression al- gorithm. Technical Report 124, Digital Equipment Corpo- ration Systems Research Center, 1994.","title":"A block sorting lossless data compression algorithm","authors":"Wheeler ; Burrows, Michael Burrows, David Wheeler","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen and Goodman, 1999] Stanley F Chen and Joshua Goodman. An empirical study of smoothing techniques for language modeling. Computer Speech & Language, 13(4):359-393, 1999.","title":"An empirical study of smoothing techniques for language modeling","authors":"Chen, Goodman ; Stanley, F Chen, Joshua Goodman","year":"1999","venue":"Computer Speech & Language","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ferragina et al., 2008] Paolo Ferragina, Rodrigo González, Gonzalo Navarro, and Rossano Venturini. Compressed text indexes: From theory to practice. ACM J. of Exp. Algorithmics, 13, 2008.","title":"Compressed text indexes: From theory to practice","authors":"Paolo Ferragina, Rodrigo González, Gonzalo Navarro, Rossano Venturini","year":"2008","venue":"ACM J. of Exp. Algorithmics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gasthaus and Teh, 2010] Jan Gasthaus and Yee W. Teh. Im- provements to the sequence memoizer. In Advances in Neural Information Processing Systems 23, pages 685- 693, 2010.","title":"Improvements to the sequence memoizer","authors":"Teh Gasthaus, Jan Gasthaus, Yee W Teh","year":"2010","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gasthaus et al., 2010] Jan Gasthaus, Frank Wood, and Yee Whye Teh. Lossless compression based on the se- quence memoizer. In 2010 Data Compression Conference (DCC 2010), pages 337-345, 2010.","title":"Lossless compression based on the sequence memoizer","authors":"Gasthaus","year":"2010","venue":"2010 Data Compression Conference (DCC 2010)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Goldwater et al., 2011] Sharon Goldwater, Thomas L Grif- fiths, and Mark Johnson. Producing power-law distribu- tions and damping word frequencies with two-stage lan- guage models. Journal of Machine Learning Research, 12(Jul):2335-2382, 2011.","title":"Producing power-law distributions and damping word frequencies with two-stage language models","authors":"Sharon Goldwater, Thomas L Griffiths, Mark Johnson","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hsu and Shiue, 1998] Leetsch C Hsu and Peter Jau-Shyong Shiue. A unified approach to generalized stirling numbers. Advances in Applied Mathematics, 20(3):366-384, 1998.","title":"A unified approach to generalized stirling numbers","authors":"Shiue ; Hsu, C Leetsch, Peter Jau-Shyong Hsu, Shiue","year":"1998","venue":"Advances in Applied Mathematics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kneser and Ney, 1995] Reinhard Kneser and Hermann Ney. Improved backing-off for m-gram language modeling. In Proceedings of IEEE International Conference on Acous- tics, Speech and Signal Processing, volume 1, pages 181- 184, 1995.","title":"Improved backing-off for m-gram language modeling","authors":"Ney Kneser, Reinhard Kneser, Hermann Ney","year":"1995","venue":"Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Koehn, 2005] Philipp Koehn. Europarl: A parallel corpus for statistical machine translation. In Proceedings of the Machine Translation summit, 2005.","title":"Europarl: A parallel corpus for statistical machine translation","authors":"Philipp Koehn","year":"2005","venue":"Proceedings of the Machine Translation summit","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Manber and Myers, 1993] Udi Manber and Eugene W. My- ers. Suffix arrays: A new method for on-line string searches. SIAM Journal on Computing, 22(5):935-948, 1993.","title":"Suffix arrays: A new method for on-line string searches","authors":"Myers ; ] Udi Manber, Eugene W Manber, Myers","year":"1993","venue":"SIAM Journal on Computing","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ohlebusch et al., 2010] Enno Ohlebusch, Johannes Fischer, and Simon Gog. CST++. In Proceedings of the Interna- tional Symposium on String Processing and Information Retrieval, 2010.","title":"CST++","authors":"Enno Ohlebusch, Johannes Fischer, Simon Gog","year":"2010","venue":"Proceedings of the International Symposium on String Processing and Information Retrieval","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Parker et al., 2011] Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. English gigaword fifth edition. Linguistic Data Consortium, (LDC2011T07), 2011.","title":"English gigaword fifth edition. Linguistic Data Consortium","authors":"Robert Parker, David Graff, Junbo Kong, Ke Chen, Kazuaki Maeda","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pitman and Yor, 1997] Jim Pitman and Marc Yor. The two- parameter Poisson-Dirichlet distribution derived from a stable subordinator. The Annals of Probability, pages 855- 900, 1997.","title":"The twoparameter Poisson-Dirichlet distribution derived from a stable subordinator","authors":"Yor Pitman, Jim Pitman, Marc Yor","year":"1997","venue":"The Annals of Probability","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Schnattinger et al., 2010] Thomas Schnattinger, Enno Ohlebusch, and Simon Gog. Bidirectional search in a string with wavelet trees. In Proceedings of the Annual Symposium on Combinatorial Pattern Matching, 2010.","title":"Bidirectional search in a string with wavelet trees","authors":"Thomas Schnattinger, Enno Ohlebusch, Simon Gog","year":"2010","venue":"Proceedings of the Annual Symposium on Combinatorial Pattern Matching","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Shareghi et al., 2015] Ehsan Shareghi, Matthias Petri, Gho- lamreza Haffari, and Trevor Cohn. Compact, efficient and unlimited capacity: Language modeling with compressed suffix trees. In Proceedings of the Conference on Empiri- cal Methods in Natural Language Processing, 2015.","title":"Compact, efficient and unlimited capacity: Language modeling with compressed suffix trees","authors":"Ehsan Shareghi, Matthias Petri, Gholamreza Haffari, Trevor Cohn","year":"2015","venue":"Proceedings of the Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Shareghi et al., 2016a] Ehsan Shareghi, Trevor Cohn, and Gholamreza Haffari. Richer interpolative smoothing based on modified kneser-ney language modeling. In Proceed- ings of the Conference on Empirical Methods in Natural Language Processing, 2016.","title":"Richer interpolative smoothing based on modified kneser-ney language modeling","authors":"Shareghi","year":"2016","venue":"Proceedings of the Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Shareghi et al., 2016b] Ehsan Shareghi, Matthias Petri, Gholamreza Haffari, and Trevor Cohn. Fast, small and exact: Infinite-order language modelling with compressed suffix trees. Transactions of the Association for Computa- tional Linguistics, 4:477-490, 2016.","title":"Fast, small and exact: Infinite-order language modelling with compressed suffix trees","authors":"Shareghi","year":"2016","venue":"Transactions of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Stolcke, 2002] Andreas Stolcke. SRILM-an extensible lan- guage modeling toolkit. In Proceedings of the Interna- tional Conference of Spoken Language Processing, 2002.","title":"SRILM-an extensible language modeling toolkit","authors":"Andreas Stolcke","year":"2002","venue":"Proceedings of the International Conference of Spoken Language Processing","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Teh et al., 2012] Yee Whye Teh, Michael I Jordan, Matthew J Beal, and David M Blei. Hierarchical dirichlet processes. Journal of the american statistical association, 2012.","title":"Hierarchical dirichlet processes","authors":"Yee Whye Teh, Michael I Jordan, Matthew J Beal, David M Blei","year":"2012","venue":"Journal of the american statistical association","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Teh, 2006a] Yee Whye Teh. A Bayesian interpretation of interpolated Kneser-Ney. Technical report, NUS School of Computing, 2006.","title":"A Bayesian interpretation of interpolated Kneser-Ney","authors":"; Teh, Yee Whye Teh","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Teh, 2006b] Yee Whye Teh. A hierarchical Bayesian lan- guage model based on Pitman-Yor processes. In Proceed- ings of the Annual Meeting of the Association for Compu- tational Linguistics, 2006.","title":"A hierarchical Bayesian language model based on Pitman-Yor processes","authors":"; Teh, Yee Whye Teh","year":"2006","venue":"Proceedings of the Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Weiner, 1973] Peter Weiner. Linear pattern matching algo- rithms. In Proceedings of the Annual Symposium Switch- ing and Automata Theory, 1973.","title":"Linear pattern matching algorithms","authors":"Peter Weiner","year":"1973","venue":"Proceedings of the Annual Symposium Switching and Automata Theory","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wood et al., 2011] Frank Wood, Jan Gasthaus, Cédric Ar- chambeau, Lancelot James, and Yee Whye Teh. The sequence memoizer. Communications of the ACM, 54(2):91-98, 2011.","title":"The sequence memoizer","authors":"Wood","year":"2011","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}