{"type":"ScholarlyArticle","title":"Fast and Accurate Neural CRF Constituency Parsing","authors":[{"name":"Yu Zhang","slug":"yu-zhang","given":"Yu","family":"Zhang","affiliation":"Institute of Artificial Intelligence , School of Computer Science and Technology , Soochow University , Suzhou , China"},{"name":"Houquan Zhou","slug":"houquan-zhou","given":"Houquan","family":"Zhou","affiliation":"Institute of Artificial Intelligence , School of Computer Science and Technology , Soochow University , Suzhou , China"},{"name":"Zhenghua Li","slug":"zhenghua-li","given":"Zhenghua","family":"Li","affiliation":"Institute of Artificial Intelligence , School of Computer Science and Technology , Soochow University , Suzhou , China"}],"abstract":"Estimating probability distribution is one of the core issues in the NLP field. \r\nHowever, in both deep learning (DL) and pre-DL eras, unlike the vast applications of linear-chain CRF in sequence labeling tasks, very few works have applied tree-structure CRF to constituency parsing, mainly due to the complexity and inefficiency of the inside-outside algorithm.\r\nThis work presents a fast and accurate neural CRF constituency parser. \r\nThe key idea is to batchify the inside algorithm for loss computation by direct large tensor operations on GPU, and meanwhile avoid the outside algorithm for gradient computation via efficient back-propagation.\r\nWe also propose a simple two-stage bracketing-then-labeling parsing approach to improve efficiency further.\r\nTo improve the parsing performance, inspired by recent progress in dependency parsing, we introduce a new scoring architecture based on boundary representation and biaffine attention, and a beneficial dropout strategy.\r\nExperiments on PTB, CTB5.1, and CTB7 show that our two-stage CRF parser achieves new state-of-the-art performance on both settings of w/o and w/ BERT, and can parse over 1,000 sentences per second.\r\nWe release our code at https://github.com/yzhangcs/crfpar.","doi":"10.24963/ijcai.2020/560","url":"https://ijcai.org/proceedings/2020/560","pdf":"https://ijcai.org/proceedings/2020/0560.pdf","markdown":"https://ijcai.org/proceedings/2020/560.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/560","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Natural Language Processing","track":"Main track","number":560,"pages":{"first":4046,"last":4053},"keywords":["Natural Language Processing: Tagging, chunking, and parsing"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24706/file_24706.m3u8","length_sec":281,"provider":"hls"}],"references":[{"position":1,"raw":"[Akoury et al., 2019] Nader Akoury, Kalpesh Krishna, and Mohit Iyyer. Syntactically supervised transformers for faster neural machine translation. In Proceedings of ACL, pages 1269-1281, 2019.","title":"Syntactically supervised transformers for faster neural machine translation","authors":"Nader Akoury, Kalpesh Krishna, Mohit Iyyer","year":"2019","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Collins, 1997] Michael Collins. Three generative, lexi- calised models for statistical parsing. In Proceedings of ACL, pages 16-23, 1997.","title":"Three generative, lexicalised models for statistical parsing","authors":"Michael Collins","year":"1997","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cross and Huang, 2016] James Cross and Liang Huang. Span-based constituency parsing with a structure-label system and provably optimal dynamic oracles. In Proceed- ings of EMNLP, pages 1-11, 2016.","title":"Span-based constituency parsing with a structure-label system and provably optimal dynamic oracles","authors":"James Cross, Liang Huang","year":"2016","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of NAACL, pages 4171-4186, 2019.","title":"BERT: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"Proceedings of NAACL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dozat and Manning, 2017] Timothy Dozat and Christo- pher D. Manning. Deep biaffine attention for neural de- pendency parsing. In Proceedings of ICLR, 2017.","title":"Deep biaffine attention for neural dependency parsing","authors":"Timothy Dozat, Christopher D Manning","year":"2017","venue":"Proceedings of ICLR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Durrett and Klein, 2015] Greg Durrett and Dan Klein. Neu- ral CRF parsing. In Proceedings of ACL-IJCNLP, pages 302-312, 2015.","title":"Neural CRF parsing","authors":"Klein Durrett, Greg Durrett, Dan Klein","year":"2015","venue":"Proceedings of ACL-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Eisner, 2016] Jason Eisner. Inside-outside and forward- backward algorithms are just backprop (tutorial paper). In Proceedings of WS, pages 1-17, 2016.","title":"Inside-outside and forwardbackward algorithms are just backprop (tutorial paper)","authors":"Jason Eisner","year":"2016","venue":"Proceedings of WS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Finkel et al., 2008] Jenny Rose Finkel, Alex Kleeman, and Christopher D. Manning. Efficient, feature-based, condi- tional random field parsing. In Proceedings of ACL, pages 959-967, 2008.","title":"Efficient, feature-based, conditional random field parsing","authors":"Jenny Rose Finkel, Alex Kleeman, Christopher D Manning","year":"2008","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gaddy et al., 2018] David Gaddy, Mitchell Stern, and Dan Klein. What's going on in neural constituency parsers? an analysis. In Proceedings of NAACL, pages 999-1010, 2018.","title":"What's going on in neural constituency parsers? an analysis","authors":"David Gaddy, Mitchell Stern, Dan Klein","year":"2018","venue":"Proceedings of NAACL","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gómez-Rodríguez and Vilares, 2018] Carlos Gómez- Rodríguez and David Vilares. Constituent parsing as sequence labeling. In Proceedings of EMNLP, pages 1314-1324, 2018.","title":"Constituent parsing as sequence labeling","authors":"Carlos Gómez, -Rodríguez, David Vilares","year":"2018","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jin et al., 2020] Lifeng Jin, Linfeng Song, Yue Zhang, Kun Xu, Wei yun Ma, and Dong Yu. Relation extraction ex- ploiting full dependency forests. In Proceedings of AAAI, 2020.","title":"Relation extraction exploiting full dependency forests","authors":"Lifeng Jin, Linfeng Song, Yue Zhang, Kun Xu, Dong Wei Yun Ma, Yu","year":"2020","venue":"Proceedings of AAAI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kaplan et al., 2004] Ron Kaplan, Stefan Riezler, Tracy H. King, John T. Maxwell III, Alex Vasserman, and Richard Crouch. Speed and accuracy in shallow and deep stochas- tic parsing. In Proceedings of HLT-NAACL, pages 97-104, 2004.","title":"Speed and accuracy in shallow and deep stochastic parsing","authors":"Ron Kaplan, Stefan Riezler, Tracy H King, John T Maxwell, Iii, Alex Vasserman, Richard Crouch","year":"2004","venue":"Proceedings of HLT-NAACL","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kitaev and Klein, 2018] Nikita Kitaev and Dan Klein. Con- stituency parsing with a self-attentive encoder. In Proceed- ings of ACL, pages 2676-2686, 2018.","title":"Constituency parsing with a self-attentive encoder","authors":"Nikita Kitaev, Dan Klein","year":"2018","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kitaev et al., 2019] Nikita Kitaev, Steven Cao, and Dan Klein. Multilingual constituency parsing with self- attention and pre-training. In Proceedings of ACL, pages 3499-3505, 2019.","title":"Multilingual constituency parsing with selfattention and pre-training","authors":"Nikita Kitaev, Steven Cao, Dan Klein","year":"2019","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lample et al., 2016] Guillaume Lample, Miguel Balles- teros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. Neural architectures for named entity recogni- tion. In Proceedings of NAACL, pages 2475-2485, 2016.","title":"Neural architectures for named entity recognition","authors":"Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, Chris Dyer","year":"2016","venue":"Proceedings of NAACL","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Le and Zuidema, 2014] Phong Le and Willem Zuidema. The inside-outside recursive neural network model for de- pendency parsing. In Proceedings of EMNLP, pages 729- 739, 2014.","title":"The inside-outside recursive neural network model for dependency parsing","authors":"Le, Zuidema ;, Phong Le, Willem Zuidema","year":"2014","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2019] Ying Li, Zhenghua Li, Min Zhang, Rui Wang, Sheng Li, and Luo Si. Self-attentive biaffine depen- dency parsing. In Proceedings of IJCAI, pages 5067-5073, 2019.","title":"Self-attentive biaffine dependency parsing","authors":"Ying Li, Zhenghua Li, Min Zhang, Rui Wang, Sheng Li, Luo Si","year":"2019","venue":"Proceedings of IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/704"},{"position":18,"raw":"[Matsuzaki et al., 2005] Takuya Matsuzaki, Yusuke Miyao, and Jun'ichi Tsujii. Probabilistic CFG with latent annota- tions. In Proceedings of ACL, pages 75-82, 2005.","title":"Probabilistic CFG with latent annotations","authors":"Matsuzaki","year":"2005","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Peters et al., 2018] Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representa- tions. In Proceedings of NAACL, pages 2227-2237, 2018.","title":"Deep contextualized word representations","authors":"Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, Luke Zettlemoyer","year":"2018","venue":"Proceedings of NAACL","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Petrov and Klein, 2007] Slav Petrov and Dan Klein. Im- proved inference for unlexicalized parsing. In Proceedings of NAACL, pages 404-411, 2007.","title":"Improved inference for unlexicalized parsing","authors":"Klein Petrov, Slav Petrov, Dan Klein","year":"2007","venue":"Proceedings of NAACL","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rush, 2020] Alexander M. Rush. Torch-struct: Deep struc- tured prediction library. arXiv:2002.00876, 2020.","title":"Torch-struct: Deep structured prediction library","authors":"Alexander M Rush","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sagae and Lavie, 2005] Kenji Sagae and Alon Lavie. A classifier-based parser with linear run-time complexity. In Proceedings of IWPT, pages 125-132, 2005.","title":"A classifier-based parser with linear run-time complexity","authors":"Kenji Sagae, Alon Lavie","year":"2005","venue":"Proceedings of IWPT","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Shen et al., 2018] Yikang Shen, Zhouhan Lin, Athul Paul Jacob, Alessandro Sordoni, Aaron Courville, and Yoshua Bengio. Straight to the tree: Constituency parsing with neural syntactic distance. In Proceedings of ACL, pages 1171-1180, 2018.","title":"Straight to the tree: Constituency parsing with neural syntactic distance","authors":"Shen","year":"2018","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Smith and Smith, 2007] David A. Smith and Noah A. Smith. Probabilistic models of nonprojective dependency trees. In Proceedings of EMNLP, pages 132-140, 2007.","title":"Probabilistic models of nonprojective dependency trees","authors":"David A Smith, Noah A Smith","year":"2007","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Stern et al., 2017] Mitchell Stern, Jacob Andreas, and Dan Klein. A minimal span-based neural constituency parser. In Proceedings of ACL, pages 818-827, 2017.","title":"A minimal span-based neural constituency parser","authors":"Mitchell Stern, Jacob Andreas, Dan Klein","year":"2017","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Taskar et al., 2004] Ben Taskar, Dan Klein, Mike Collins, Daphne Koller, and Christopher Manning. Max-margin parsing. In Proceedings of EMNLP, pages 1-8, 2004.","title":"Max-margin parsing","authors":"Ben Taskar, Dan Klein, Mike Collins, Daphne Koller, Christopher Manning","year":"2004","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Teng and Zhang, 2018] Zhiyang Teng and Yue Zhang. Two local models for neural constituent parsing. In Proceed- ings of COLING, pages 119-132, 2018.","title":"Two local models for neural constituent parsing","authors":"Zhiyang Teng, Yue Zhang","year":"2018","venue":"Proceedings of COLING","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Vilares et al., 2019] David Vilares, Mostafa Abdou, and Anders Søgaard. Better, faster, stronger sequence tag- ging constituent parsers. In Proceedings of NAACL, pages 3372-3383, 2019.","title":"Better, faster, stronger sequence tagging constituent parsers","authors":"David Vilares, Mostafa Abdou, Anders Søgaard","year":"2019","venue":"Proceedings of NAACL","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang and Chang, 2016] Wenhui Wang and Baobao Chang. Graph-based dependency parsing with bidirectional LSTM. In Proceedings of ACL, pages 2475-2485, 2016.","title":"Graph-based dependency parsing with bidirectional LSTM","authors":"Wang, Chang ; ] Wenhui Wang, Baobao Chang","year":"2016","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2018] Xinyi Wang, Hieu Pham, Pengcheng Yin, and Graham Neubig. A tree-based decoder for neu- ral machine translation. In Proceedings of EMNLP, pages 4772-4777, 2018.","title":"A tree-based decoder for neural machine translation","authors":"Xinyi Wang, Hieu Pham, Pengcheng Yin, Graham Neubig","year":"2018","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang et al., 2020] Yu Zhang, Zhenghua Li, and Zhang Min. Efficient second-order TreeCRF for neural depen- dency parsing. In Proceedings of ACL, 2020.","title":"Efficient second-order TreeCRF for neural dependency parsing","authors":"Yu Zhang, Zhenghua Li, Zhang Min","year":"2020","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhou and Zhao, 2019] Junru Zhou and Hai Zhao. Head- driven phrase structure grammar parsing on Penn treebank. In Proceedings of the ACL, pages 2396-2408, 2019.","title":"Headdriven phrase structure grammar parsing on Penn treebank","authors":"Zhao ; ] Junru Zhou, Hai Zhou, Zhao","year":"2019","venue":"Proceedings of the ACL","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhu et al., 2013] Muhua Zhu, Yue Zhang, Wenliang Chen, Min Zhang, and Jingbo Zhu. Fast and accurate shift-reduce constituent parsing. In Proceedings of ACL, pages 434- 443, 2013.","title":"Fast and accurate shift-reduce constituent parsing","authors":"Muhua Zhu, Yue Zhang, Wenliang Chen, Min Zhang, Jingbo Zhu","year":"2013","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}