{"type":"ScholarlyArticle","title":"Specializing Word Embeddings (for Parsing) by Information Bottleneck (Extended Abstract)","authors":[{"name":"Xiang Lisa Li","slug":"xiang-lisa-li","given":"Xiang Lisa","family":"Li","affiliation":"Department of Computer Science Johns Hopkins University"},{"name":"Jason Eisner","slug":"jason-eisner","given":"Jason","family":"Eisner","affiliation":"Department of Computer Science Johns Hopkins University"}],"abstract":"Pre-trained word embeddings like ELMo and BERT contain rich syntactic and semantic information, resulting in state-of-the-art performance on various tasks. We propose a very fast variational information bottleneck (VIB) method to nonlinearly compress these embeddings, keeping only the information that helps a discriminative parser. We compress each word embedding to either a discrete tag or a continuous vector. In the discrete version, our automatically compressed tags form an alternative tag set: we show experimentally that our tags capture most of the information in traditional POS tag annotations, but our tag sequences can be parsed more accurately at the same level of tag granularity. In the continuous version, we show experimentally that moderately compressing the word embeddings by our method yields a more accurate parser in 8 of 9 languages, unlike simple dimensionality reduction.","doi":"10.24963/ijcai.2020/658","url":"https://ijcai.org/proceedings/2020/658","pdf":"https://ijcai.org/proceedings/2020/0658.pdf","markdown":"https://ijcai.org/proceedings/2020/658.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/658","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Sister Conferences Best Papers","track":"Sister Conferences Best Papers","number":658,"pages":{"first":4745,"last":4749},"keywords":["Natural Language Processing: Tagging, chunking, and parsing","Natural Language Processing: Embeddings","Natural Language Processing: Natural Language Processing"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25539/file_25539.m3u8","length_sec":297,"provider":"hls"}],"references":[{"position":1,"raw":"[Alemi et al., 2016] Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy. Deep variational infor- mation bottleneck. Proceedings of the International Confer- ence on Learning Representations (ICLR), abs/1612.00410, 2016.","title":"Deep variational information bottleneck","authors":"Alexander A Alemi, Ian Fischer, Joshua V Dillon, Kevin Murphy","year":"2016","venue":"Proceedings of the International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Anderson, 2003] T.W. Anderson. An Introduction to Multi- variate Statistical Analysis. Wiley Series in Probability and Statistics. Wiley, 2003.","title":"An Introduction to Multivariate Statistical Analysis","authors":"T W Anderson","year":"2003","venue":"Wiley Series in Probability and Statistics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Che et al., 2018] Wanxiang Che, Yijia Liu, Yuxuan Wang, Bo Zheng, and Ting Liu. Towards better UD parsing: Deep contextualized word embeddings, ensemble, and treebank concatenation. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 55-64, Brussels, Belgium, October 2018. Association for Computational Linguistics.","title":"Towards better UD parsing: Deep contextualized word embeddings, ensemble, and treebank concatenation","authors":"Wanxiang Che, Yijia Liu, Yuxuan Wang, Bo Zheng, Ting Liu","year":"2018","venue":"Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018.","title":"BERT: pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dozat and Manning, 2016] Timothy Dozat and Christo- pher D. Manning. Deep biaffine attention for neural depen- dency parsing. CoRR, abs/1611.01734, 2016.","title":"Deep biaffine attention for neural dependency parsing","authors":"Timothy Dozat, Christopher D Manning","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gardner et al., 2017] Matt Gardner, Joel Grus, Mark Neu- mann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke S. Zettlemoyer. AllenNLP: A Deep Semantic Natural Language Processing Platform. 2017.","title":"AllenNLP: A Deep Semantic Natural Language Processing Platform","authors":"Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F Liu, Matthew Peters, Michael Schmitz, Luke S Zettlemoyer","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Goldberg, 2019] Yoav Goldberg. Assessing BERT's syntac- tic abilities. CoRR, abs/1901.05287, 2019.","title":"Assessing BERT's syntactic abilities","authors":"Yoav Goldberg","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kingma and Ba, 2014] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceed- ings of the International Conference on Learning Represen- tations (ICLR), 2014.","title":"Adam: A method for stochastic optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2014","venue":"Proceedings of the International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Klein and Manning, 2003] D. Klein and C. D. Manning. Ac- curate unlexicalized parsing. In Proceedings of the 41st Annual Meeting of the Association for Computational Lin- guistics, 2003.","title":"Accurate unlexicalized parsing","authors":"D Klein, C D Manning","year":"2003","venue":"Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li and Eisner, 2019] Xiang Lisa Li and Jason Eisner. Spe- cializing word embeddings (for parsing) by information bottleneck. In Proceedings of the 2019 Conference on Em- pirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Pro- cessing, pages 2744-2754, Hong Kong, November 2019. Best Paper Award.","title":"Specializing word embeddings (for parsing) by information bottleneck","authors":"Lisa Xiang, Jason Li, Eisner","year":"2019","venue":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Nivre et al., 2018] Joakim Nivre et al. Universal dependen- cies 2.3, 2018. LINDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics ( ÚFAL), Fac- ulty of Mathematics and Physics, Charles University.","title":"LINDAT/CLARIN digital library","authors":"Nivre","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Peters et al., 2018a] Matthew Peters, Mark Neumann, Mo- hit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word repre- sentations. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 1 (Long Papers), 2018.","title":"Deep contextualized word representations","authors":"Peters","year":"2018","venue":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Peters et al., 2018b] Matthew E. Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. Dissecting contextual word embeddings: Architecture and representation. In Pro- ceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1499-1509, 2018.","title":"Dissecting contextual word embeddings: Architecture and representation","authors":"Peters","year":"2018","venue":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Tishby and Zaslavsky, 2015] Naftali Tishby and Noga Za- slavsky. Deep learning and the information bottleneck principle. 2015 IEEE Information Theory Workshop (ITW), pages 1-5, 2015.","title":"Deep learning and the information bottleneck principle","authors":"Naftali Tishby, Noga Zaslavsky","year":"2015","venue":"IEEE Information Theory Workshop (ITW)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Tishby et al., 2000] Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.","title":"The information bottleneck method","authors":"Naftali Tishby, Fernando C Pereira, William Bialek","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}