Proceedings

A Dependency-based Method for Evaluating Broad-Coverage Parsers

Download BibTeX Cite

Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence. Natural Language. Pages 1420-1426.

© 1995 International Joint Conferences on Artificial Intelligence

Abstract

With the emergence of broad-coverage parsers, quantitative evaluation of parsers becomes increasingly more important. We propose a dependency-based method for evaluating broad-coverage parsers. The method offers several advantages over previous methods that are based on phrase boundaries. The error count score we propose here is not only more intuitively meaningful than other scores, but also more relevant to semantic interpretation. We will also present an algorithm for transforming constituency trees into dependency trees so that the evaluation method is applicable to both dependency and constituency grammars . Finally, we discuss a set of operations for modifying dependency trees that can be used to eliminate inconsequential differences among different parse trees and allow us to selectively evalu ate different aspects of a parser.

  1. [Black et a/., 1991] E. Black, S. Abney, D . Flickenger, C. Gdaniec, R. Grishman , P. Harrison , D . Hindle, R . Ingria, F . Jelinek , J. Klavans , M. Liberman, M. Marcus, S. Roukos , B Santorini , and T. Strza- lkowski . A procedure for quantitatively comparing the syntactic coverage of english grammars . In Pro- ceedings of Speech and Natural Languag e Workshop, pages 306-311. DARPA , February 1991.
  2. [Black et a/., 1992] Ezra Black, John Lafferty, and Salim Roukos . Development and evaluation of a broad- coverage probabilistic grammar of English-language computer manuals . In Proceedings of ACL-92, pages 185-192, Newark, Delaware, 1992.
  3. [Magerman , 1994] David M. Magerman. Natural Lan- guage Parsing as Statistical Pattern R ecognition. PhD thesis , Stanford University, 1994.
  4. [Mel 'cuk , 1987] Igor A . Mel'cuk . D ependency syntax: theory and pra ctice. State University of New York Press, Albany, 1987.

References were extracted automatically from the PDF and may contain errors.

Welcome Future Conferences AI Journal Division IJCAI Proceedings About IJCAI IJCAI Awards