{"type":"ScholarlyArticle","title":"Unsupervised and Few-Shot Parsing from Pretrained Language Models (Extended Abstract)","authors":[{"name":"Zhiyuan Zeng","slug":"zhiyuan-zeng","given":"Zhiyuan","family":"Zeng","affiliation":"Tianjin University"},{"name":"Deyi Xiong","slug":"deyi-xiong","given":"Deyi","family":"Xiong","affiliation":"Tianjin University"}],"abstract":"This paper proposes two Unsupervised constituent Parsing models (UPOA and UPIO) that calculate inside and outside association scores solely based on the self-attention weight matrix learned in a pretrained language model. The proposed unsupervised parsing models are further extended to few-shot parsing models (FPOA, FPIO) that use a few annotated trees to fine-tune the linear projection matrices in self-attention. Experiments on PTB and SPRML show that both unsupervised and few-shot parsing methods are better than or comparable to the previous methods.","doi":"10.24963/ijcai.2023/797","url":"https://ijcai.org/proceedings/2023/797","pdf":"https://ijcai.org/proceedings/2023/0797.pdf","markdown":"https://ijcai.org/proceedings/2023/797.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/797","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Journal Track","track":"Journal Track","number":797,"pages":{"first":6995,"last":7000},"keywords":["Natural Language Processing: General","Natural Language Processing: NLP: Tagging, chunking, and parsing"],"videos":[],"references":[{"position":1,"raw":"[Dahlmeier et al., 2013] Daniel Dahlmeier, Hwee Tou Ng, and Siew Mei Wu. Building a large annotated corpus of learner english: The NUS corpus of learner english.","title":"Building a large annotated corpus of learner english: The NUS corpus of learner english","authors":"Daniel Dahlmeier, Hwee Tou Ng, Mei Siew, Wu","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"In Joel R. Tetreault, Jill Burstein, and Claudia Leacock, editors, Proceedings of the Eighth Workshop on Innova- tive Use of NLP for Building Educational Applications, BEA@NAACL-HLT 2013, June 13, 2013, Atlanta, Geor- gia, USA, pages 22-31. The Association for Computer Linguistics, 2013.","title":"Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications, BEA@NAACL-HLT 2013","authors":null,"year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understand- ing. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics, 2019.","title":"BERT: pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Drozdov et al., 2019] Andrew Drozdov, Patrick Verga, Mo- hit Yadav, Mohit Iyyer, and Andrew McCallum. Unsuper- vised latent tree induction with deep inside-outside recur- sive auto-encoders. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2- 7, 2019, Volume 1 (Long and Short Papers), pages 1129- 1141. Association for Computational Linguistics, 2019.","title":"Unsupervised latent tree induction with deep inside-outside recursive auto-encoders","authors":"Andrew Drozdov, Patrick Verga, Mohit Yadav, Mohit Iyyer, Andrew Mccallum","year":"2019","venue":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Jawahar et al., 2019] Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What does BERT learn about the struc- ture of language? In Anna Korhonen, David R. Traum, and Lluís Màrquez, editors, Proceedings of the 57th Con- ference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Vol- ume 1: Long Papers, pages 3651-3657. Association for Computational Linguistics, 2019.","title":"What does BERT learn about the structure of language?","authors":"Ganesh Jawahar, Benoît Sagot, Djamé Seddah","year":"2019","venue":"Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kim et al., 2019a] Yoon Kim, Chris Dyer, and Alexan- der M. Rush. Compound probabilistic context-free gram- mars for grammar induction. In Anna Korhonen, David R. Traum, and Lluís Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Lin- guistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pages 2369-2385. Asso- ciation for Computational Linguistics, 2019.","title":"Compound probabilistic context-free grammars for grammar induction","authors":"Kim","year":"2019","venue":"Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kim et al., 2019b] Yoon Kim, Alexander M. Rush, Lei Yu, Adhiguna Kuncoro, Chris Dyer, and Gábor Melis. Un- supervised recurrent neural network grammars. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Pro- ceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Min- neapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 1105-1117. Association for Compu- tational Linguistics, 2019.","title":"Unsupervised recurrent neural network grammars","authors":"Kim","year":"2019","venue":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kim et al., 2020a] Taeuk Kim, Jihun Choi, Daniel Edmis- ton, and Sang-goo Lee. Are pre-trained language models aware of phrases? simple but strong baselines for gram- mar induction. In 8th International Conference on Learn- ing Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.","title":"Are pre-trained language models aware of phrases? simple but strong baselines for grammar induction","authors":"Kim","year":"2020","venue":"8th International Conference on Learn-ing Representations, ICLR 2020","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kim et al., 2020b] Taeuk Kim, B. Li, and Sanggoo Lee. Chart-based zero-shot constituency parsing on multiple languages. arXiv: Computation and Language, 2020.","title":"Chart-based zero-shot constituency parsing on multiple languages","authors":"Kim","year":"2020","venue":"arXiv: Computation and Language","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kitaev et al., 2019] Nikita Kitaev, Steven Cao, and Dan Klein. Multilingual constituency parsing with self- attention and pre-training. In Anna Korhonen, David R. Traum, and Lluís Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Lin- guistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pages 3499-3505. Asso- ciation for Computational Linguistics, 2019.","title":"Multilingual constituency parsing with selfattention and pre-training","authors":"Nikita Kitaev, Steven Cao, Dan Klein","year":"2019","venue":"Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692, 2019.","title":"Roberta: A robustly optimized BERT pretraining approach","authors":"Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Radford et al., 2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9, 2019.","title":"Language models are unsupervised multitask learners","authors":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever","year":"2019","venue":"OpenAI Blog","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Seddah and others, 2013] Djamé Seddah et al. Overview of the SPMRL 2013 shared task: A cross-framework eval- uation of parsing morphologically rich languages. In","title":"Djamé Seddah et al. Overview of the SPMRL 2013 shared task: A cross-framework evaluation of parsing morphologically rich languages","authors":"Others Seddah","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"Yoav Goldberg, Yuval Marton, Ines Rehbein, and Yannick Versley, editors, Proceedings of the Fourth Workshop on Statistical Parsing of Morphologically-Rich Languages, SPMRL@EMNLP 2013, Seattle, Washington, USA, Octo- ber 18, 2013, pages 146-182. Association for Computa- tional Linguistics, 2013.","title":"Proceedings of the Fourth Workshop on Statistical Parsing of Morphologically-Rich Languages, SPMRL@EMNLP 2013","authors":null,"year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Shen et al., 2018a] Yikang Shen, Zhouhan Lin, Chin-Wei Huang, and Aaron C. Courville. Neural language mod- eling by jointly learning syntax and lexicon. In 6th Inter- national Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 -May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.","title":"Neural language modeling by jointly learning syntax and lexicon","authors":"Shen","year":"2018","venue":"6th International Conference on Learning Representations, ICLR 2018","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Shen et al., 2018b] Yikang Shen, Zhouhan Lin, Athul Paul Jacob, Alessandro Sordoni, Aaron C. Courville, and Yoshua Bengio. Straight to the tree: Constituency pars- ing with neural syntactic distance. In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Vol- ume 1: Long Papers, pages 1171-1180. Association for Computational Linguistics, 2018.","title":"Straight to the tree: Constituency parsing with neural syntactic distance","authors":"Shen","year":"2018","venue":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Shen et al., 2019] Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron C. Courville. Ordered neurons: In- tegrating tree structures into recurrent neural networks. In 7th International Conference on Learning Representa- tions, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.","title":"Ordered neurons: Integrating tree structures into recurrent neural networks","authors":"Shen","year":"2019","venue":"7th International Conference on Learning Representations, ICLR 2019","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Shi et al., 2020] Haoyue Shi, Karen Livescu, and Kevin Gimpel. On the role of supervision in unsupervised con- stituency parsing. In Bonnie Webber, Trevor Cohn, Yulan Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence (IJCAI-23) Journal Track He, and Yang Liu, editors, Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Pro- cessing, EMNLP 2020, Online, November 16-20, 2020, pages 7611-7621. Association for Computational Linguis- tics, 2020.","title":"On the role of supervision in unsupervised constituency parsing","authors":"Shi","year":"2020","venue":"Yulan Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence (IJCAI-23) Journal Track","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tenney et al., 2019] Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Anna Korhonen, David R. Traum, and Lluís Màrquez, ed- itors, Proceedings of the 57th Conference of the Associa- tion for Computational Linguistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pages 4593-4601. Association for Computational Linguis- tics, 2019.","title":"BERT rediscovers the classical NLP pipeline","authors":"Ian Tenney, Dipanjan Das, Ellie Pavlick","year":"2019","venue":"Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2019] Yau-Shian Wang, Hung-yi Lee, and Yun-Nung Chen. Tree transformer: Integrating tree struc- tures into self-attention. In Kentaro Inui, Jing Jiang, Vin- cent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Confer- ence on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 1061-1070. Association for Computational Linguistics, 2019.","title":"Tree transformer: Integrating tree structures into self-attention","authors":"Yau-Shian, Hung-Yi Wang, Yun-Nung Lee, Chen","year":"2019","venue":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zeng and Xiong, 2022] Zhiyuan Zeng and Deyi Xiong. Un- supervised and few-shot parsing from pretrained language models. Artif. Intell., 305:103665, 2022.","title":"Unsupervised and few-shot parsing from pretrained language models","authors":"Xiong Zeng, Zhiyuan Zeng, Deyi Xiong","year":"2022","venue":"Artif. Intell","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}