{"type":"ScholarlyArticle","title":"Self-attentive Biaffine Dependency Parsing","authors":[{"name":"Ying Li","slug":"ying-li","given":"Ying","family":"Li","affiliation":"Institute of Artificial Intelligence , School of Computer Science and Technology , Soochow University"},{"name":"Zhenghua Li","slug":"zhenghua-li","given":"Zhenghua","family":"Li","affiliation":"Institute of Artificial Intelligence , School of Computer Science and Technology , Soochow University"},{"name":"Min Zhang","slug":"min-zhang","given":"Min","family":"Zhang","affiliation":"Institute of Artificial Intelligence , School of Computer Science and Technology , Soochow University"},{"name":"Rui Wang","slug":"rui-wang","given":"Rui","family":"Wang","affiliation":"Alibaba Group , China"},{"name":"Sheng Li","slug":"sheng-li","given":"Sheng","family":"Li","affiliation":"Institute of Artificial Intelligence , School of Computer Science and Technology , Soochow University"},{"name":"Luo Si","slug":"luo-si","given":"Luo","family":"Si","affiliation":"Alibaba Group , China"}],"abstract":"The current state-of-the-art dependency parsing approaches employ BiLSTMs to encode input sentences.Motivated by the success of the transformer-based machine translation, this work for the first time applies the self-attention mechanism to dependency parsing as the replacement of the BiLSTM-based encoders, leading to competitive performance on both English and Chinese benchmark data. Based on the detailed error analysis, we then combine the power of both BiLSTM and self-attention via model ensembles, demonstrating their complementary capability of capturing contextual information. Finally, we explore the recently proposed contextualized word representations as extra input features, and further improve the parsing performance.","doi":"10.24963/ijcai.2019/704","url":"https://ijcai.org/proceedings/2019/704","pdf":"https://ijcai.org/proceedings/2019/0704.pdf","markdown":"https://ijcai.org/proceedings/2019/704.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/704","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Natural Language Processing","track":"Main track","number":704,"pages":{"first":5067,"last":5073},"keywords":["Natural Language Processing: Natural Language Processing","Natural Language Processing: Tagging, chunking, and parsing"],"videos":[],"references":[{"position":1,"raw":"[Andor et al., 2016] Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. Globally nor- malized transition-based neural networks. In Proceedings of ACL, pages 2442-2452, 2016.","title":"Globally normalized transition-based neural networks","authors":"Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, Michael Collins","year":"2016","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ba et al., 2016] Lei Jimmy Ba, Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. CoRR, abs/1607.06450, 2016.","title":"Layer normalization","authors":"Jimmy Lei, Ryan Ba, Geoffrey E Kiros, Hinton","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen and Manning, 2014] Danqi Chen and Christopher D. Manning. A fast and accurate dependency parser using neural networks. In Proceedings of EMNLP, pages 740- 750, 2014.","title":"A fast and accurate dependency parser using neural networks","authors":"Chen, Manning, Danqi Chen, Christopher D Manning","year":"2014","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Clark et al., 2018] Kevin Clark, Minh-Thang Luong, Christopher D. Manning, and Quoc V. Le. Semi- supervised sequence modeling with cross-view training. In Proceedings of EMNLP, pages 1914-1925, 2018.","title":"Semisupervised sequence modeling with cross-view training","authors":"Kevin Clark, Minh-Thang Luong, Christopher D Manning, V Quoc, Le","year":"2018","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dozat and Manning, 2017] Timothy Dozat and Christopher Manning. Deep biaffine attention for neural dependency parsing. In Proceedings of ICLR, 2017.","title":"Timothy Dozat and Christopher Manning. Deep biaffine attention for neural dependency parsing","authors":"Manning Dozat","year":"2017","venue":"Proceedings of ICLR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dyer et al., 2015] Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. Transition- based dependency parsing with stack long short-term memory. In Proceedings of ACL, pages 334-343, 2015.","title":"Transitionbased dependency parsing with stack long short-term memory","authors":"Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, Noah A Smith","year":"2015","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Foster et al., 2018] George Foster, Ashish Vaswani, Jakob Uszkoreit, Wolfgang Macherey, Lukasz Kaiser, Orhan Fi- rat, Llion Jones, Noam Shazeer, Yonghui Wu, Ankur Bapna, Melvin Johnson, Mike Schuster, Zhifeng Chen, Macduff Hughes, Niki Parmar, and Mia Xu Chen. The best of both worlds: Combining recent advances in neural machine translation. In Proceedings of ACL, pages 76-86, 2018.","title":"The best of both worlds: Combining recent advances in neural machine translation","authors":"George Foster, Ashish Vaswani, Jakob Uszkoreit, Wolfgang Macherey, Lukasz Kaiser, Orhan Firat, Llion Jones, Noam Shazeer, Yonghui Wu, Ankur Bapna, Melvin Johnson, Mike Schuster, Zhifeng Chen, Macduff Hughes, Niki Parmar, Mia Xu, Chen","year":"2018","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hajic et al., 2009] Jan Hajic, Massimiliano Ciaramita, Richard Johansson, Daisuke Kawahara, Maria Antònia Martí, Lluís Màrquez, Adam Meyers, Joakim Nivre, Sebastian Padó, Jan Stepánek, Pavel Stranák, Mihai Surdeanu, Nianwen Xue, and Yi Zhang. The conll-2009 shared task: Syntactic and semantic dependencies in multiple languages. In Proceedings of CoNLL: Shared Task, pages 1-18, 2009.","title":"The conll-2009 shared task: Syntactic and semantic dependencies in multiple languages","authors":"Jan Hajic, Massimiliano Ciaramita, Richard Johansson, Daisuke Kawahara, Maria Antònia Martí, Lluís Màrquez, Adam Meyers, Joakim Nivre, Sebastian Padó, Jan Stepánek, Pavel Stranák, Mihai Surdeanu, Nianwen Xue, Yi Zhang","year":"2009","venue":"Proceedings of CoNLL: Shared Task","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hinton, 2002] Geoffrey E. Hinton. Training products of ex- perts by minimizing contrastive divergence. Neural Com- putation, 14(8):1771-1800, 2002.","title":"Training products of experts by minimizing contrastive divergence","authors":"Geoffrey E Hinton","year":"2002","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kiperwasser and Goldberg, 2016] Eliyahu Kiperwasser and Yoav Goldberg. Simple and accurate dependency parsing using bidirectional LSTM feature representations. CoRR, abs/1603.04351, 2016.","title":"Eliyahu Kiperwasser and Yoav Goldberg. Simple and accurate dependency parsing using bidirectional LSTM feature representations","authors":"Goldberg Kiperwasser","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kitaev and Klein, 2018] Nikita Kitaev and Dan Klein. Con- stituency parsing with a self-attentive encoder. In Proceed- ings of ACL, pages 2675-2685, 2018.","title":"Constituency parsing with a self-attentive encoder","authors":"Nikita Kitaev, Dan Klein","year":"2018","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kuncoro et al., 2016] Adhiguna Kuncoro, Miguel Balles- teros, Lingpeng Kong, Chris Dyer, and Noah A. Smith. Distilling an ensemble of greedy dependency parsers into one MST parser. In Proceedings of EMNLP, pages 1744- 1753, 2016.","title":"Distilling an ensemble of greedy dependency parsers into one MST parser","authors":"Kuncoro","year":"2016","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2018] Yijia Liu, Wanxiang Che, Huaipeng Zhao, Bing Qin, and Ting Liu. Distilling knowledge for search- based structured prediction. In Proceedings of ACL, pages 1393-1402, 2018.","title":"Distilling knowledge for searchbased structured prediction","authors":"Yijia Liu, Wanxiang Che, Huaipeng Zhao, Bing Qin, Ting Liu","year":"2018","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ma et al., 2018] Xuezhe Ma, Zecong Hu, Jingzhou Liu, Nanyun Peng, Graham Neubig, and Eduard H. Hovy. Stack-pointer networks for dependency parsing. In Pro- ceedings of ACL, pages 1403-1414, 2018.","title":"Stack-pointer networks for dependency parsing","authors":"Ma","year":"2018","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[McDonald et al., 2005] Ryan T. McDonald, Fernando Pereira, Kiril Ribarov, and Jan Hajic. Non-projective dependency parsing using spanning tree algorithms. In Proceedings of HLT/EMNLP, pages 523-530, 2005.","title":"Non-projective dependency parsing using spanning tree algorithms","authors":"Ryan T Mcdonald, Fernando Pereira, Kiril Ribarov, Jan Hajic","year":"2005","venue":"Proceedings of HLT/EMNLP","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Nivre and McDonald, 2008] Joakim Nivre and Ryan Mc- Donald. Integrating graph-based and transition-based de- pendency parsers. In Proceedings of ACL, pages 950-958, 2008.","title":"Integrating graph-based and transition-based dependency parsers","authors":"Mcdonald Nivre, Joakim Nivre, Ryan Mc-Donald","year":"2008","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Peters et al., 2018] Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of NAACL-HLT, pages 2227-2237, 2018.","title":"Deep contextualized word representations","authors":"Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, Luke Zettlemoyer","year":"2018","venue":"Proceedings of NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Petrov and McDonald, 2012] Slav Petrov and Ryan Mc- Donald. Overview of the 2012 shared task on parsing the web. In Proceedings of SANCL, 2012.","title":"Overview of the 2012 shared task on parsing the web","authors":"Mcdonald Petrov, Slav Petrov, Ryan Mc-Donald","year":"2012","venue":"Proceedings of SANCL","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Radford et al., 2018] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. OpenAI Techni- cal Report, 2018.","title":"Improving language understanding by generative pre-training","authors":"Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shen et al., 2018] Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan, and Chengqi Zhang. Disan: Di- rectional self-attention network for rnn/cnn-free language understanding. In Proceedings of AAAI, pages 5446-5455, 2018.","title":"Disan: Directional self-attention network for rnn/cnn-free language understanding","authors":"Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan, Chengqi Zhang","year":"2018","venue":"Proceedings of AAAI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Stern et al., 2017] Mitchell Stern, Jacob Andreas, and Dan Klein. A minimal span-based neural constituency parser. In Proceedings of ACL, pages 818-827, 2017.","title":"A minimal span-based neural constituency parser","authors":"Mitchell Stern, Jacob Andreas, Dan Klein","year":"2017","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tan et al., 2018] Zhixing Tan, Mingxuan Wang, Jun Xie, Yidong Chen, and Xiaodong Shi. Deep semantic role la- beling with self-attention. In Proceedings of AAAI, pages 4929-4936, 2018.","title":"Deep semantic role labeling with self-attention","authors":"Zhixing Tan, Mingxuan Wang, Jun Xie, Yidong Chen, Xiaodong Shi","year":"2018","venue":"Proceedings of AAAI","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of NIPS, pages 6000-6010, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Proceedings of NIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zeman et al., 2018] Daniel Zeman, Jan Haji{č}, Martin Popel, Martin Potthast, Milan Straka, Filip Ginter, Joakim Nivre, and Slav Petrov. Conll 2018 shared task: Multilin- gual parsing from raw text to universal dependencies. In Proceedings of CoNLL, pages 1-21, 2018.","title":"Conll 2018 shared task: Multilingual parsing from raw text to universal dependencies","authors":"Jan Daniel Zeman, Martin Haji{č}, Martin Popel, Milan Potthast, Filip Straka, Joakim Ginter, Slav Nivre, Petrov","year":"2018","venue":"Proceedings of CoNLL","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}