Experience with the Evaluation of Natural Language Question Answerers

Experience with the Evaluation of Natural Language Question Answerers·

Harry Tennant Advanced Automation Group Coordinated Science Laboratory University of Illinois at Urbana-Champaign Urbana, IL 61g01

Research in natural language processing could be facilitated by thorough and critical evaluations of natural language systems. Two measurements, conceptual and lingui s tic completeness, are defined and discussed in this paper. Testing done on two natural language question answerers demonstrated that the conceptual coverage of such systems should be extended to better satisfy the needs and expectations of users.

None of the natural language papers presented at IJCAI77 reported on their programs' performance as language understanders. The reader has very little hope of thoroughly understanding the capabilities of the systems and techniques described.

The most critical dimensions of natural language performance are the range of concepts understood and the accommodation of diverse forms of expression. In this work on evaluation I have concentrated on these two aspects of performance and restricted the study to natural language question answering systems. In the p!iragraphs that follow, I wi 11 describe performance goals and measurement in more detail, then discuss the

findings to which their use has led .

1.0 Performance Goals and Measurement

A natural 1 anguage processor can be seen from two points of view: that of the designer and that of the user . The designer examines the domain of discourse of a particular application and maps out the range of concepts that are withi n it. The range of concepts is the CONCEPTUAL COVERAGE of the system. The designer must also accommodate the diversity with which the users will generate their statements and requests. The range of linguistic phenomena handled by the system is the LINGUISTIC COVERAGE of the system. The gauge of the success of the designers is how well the coverage has

anticipated the needs of the users.

'This work was supported by the Office of Naval

Research under Contract N00014-75-C-061~.

Users see a natural language question answering system and the database to which i t int erfaces through the perspective of their own needs and habits. They approach a database questi on answerer expecting it to include certa i n concepts, and forming utterances in their accustomed ways. The degree to which t he concepts expected by a set of users can actuall y be found in the system's conceptual coverage is the CONCEPTUAL COMPLETENESS of t he nat ural language processor, with respect to the set of users. Similarly, the degree to wh i ch t he language of a set of users is appropri ate l y analyzed by the system is the LINGUISTIC

COMPLETENESS with respect to that set of users .

It is interesting to consider the current description techniques for natural language processors in terms of conceptual and l i ngui s tic coverage and completeness. When a paper presents twenty or so questions t hat ar e appropriately analyzed, the questions i nc l ude concepts from the system's conceptual coverage, and their forms and features suggest elements of the system's linguistic coverage. Unfortunately, the conceptual and lingui s tic coverage of the system is not fully specified by what is found i n the examples, and one canno t generalize from them. If no claim is made t ha t the examples were in some sense typical of user inputs, nothing can be inferred about conceptual or linguistic completeness. If the paper goes on to explicitly mention handling feat ures such as ellipsis or pronoun reference a comment i s being made about the elements of linguisti c coverage. Si nee this is not being rela t ed t o the needs of a set of users, it says nothing

about linguistic completeness.

2.0 Performance Testing

The i deas for testing conceptual and linguistic completeness were exercised on two natural language question answering systems, PLANE:S[9] and, to a lesser extent, the Automatic Advi sor[7]. The performance of PLANE:S will be described in (8].

One t ests the general competence of the natural language system by simply giving database probl ems to users and letting them attempt to solve them. Conceptual completeness is tested by giving database problems to subjects who are very familiar with the system, eg., the system designers. If a problem can be solved at all, the conceptual coverage must include the concepts indicated by the problem. Linguistic completeness is tested by giving users the probl ems that were solved in the conceptual completeness test. If a user utterance cannot be handled appropriately by the system, it is probably due to a limitation of linguistic coverage. It could, however, be due to a limitation of the conceptual coverage that was not di scovered in the conceptual completeness test. For thi.s reason, tests for conceptual completeness and 11 ngui.stic completeness are not entirel y independent.

3.0 Conceptual Completeness

Much work has been done and discussed in the literature on identifying the elements of lingui stic coverage for improved linguistic completeness. The user testing has not led to the discovery of new elements of linguistic cover age.

The most interesting findings were in the area of conceptual completeness. Papers describing natural language processors frequently concent rate on linguistic coverage with the implicit assumption that the database determines the range of conceptual coverage. However, we found that the subjects thought in terms of the activiti es that the database described. The acti vities orientation is evidently the one that should be supported for casual users.

The activities orientation is helpful for identi fying a broad class of erroneous presuppositions that may be embedded in user's queries. In the question "Which planes that flew on March 2, 1970 had maintenance on March 3?", one presupposition is that there were in fact some planes that flew on March 2. If the database lists the planes that flew on particular days, this would be a Kaplan-type presupposition (2]. However, if there was no information on flights in the database, it would be a more general form of presupposition.

Another result relating to conceptual completeness was evidence against the common assumption that users' utterances are limited to database queries. Most natural language question answerers rely on the assumption that the user will ask a question, the system will query the database to answer it, then the user will ask another and so on. This is not the way conversations work. In testing PLANE:S and the Automati.c Advisor, users frequently made utterances that were not intended to be interpreted as database queries. Various examples follow.

1. About the database:
2. "~at do you know"?
3. "Is that everything you know about math 1957"
4. About vocabulary:
5. "~at is abuser?"
6. 2 . "What does howmal mean?"
7. Context setting:
8. "Now I am talking about the year 1970".
9. "Year is 1970".
10. "I am interested in a7's".
11. Reference to discourse objects:
12. "What was the last plane I talked about?"
13. "Now combine the two"
14. "The other ones I just mentioned"
15. Verifying or summarizing results:
16. "Then infe 210 and infe 21 1 must be taken at the same time".
17. "So all I need is junior standing and math 195".
18. Multiple query utterances:
19. "What parts failed and what were the parts removed or installed".
20. ~ultiple utterance queries:
21. "How many aircraft flew more than 10 flights in 1973? in 1972? in 1970? List aircraft by number of flights for 1970."

Each of these examples was actually typed in by users using one of the natural language processors. Many more examples in each of these categories could be imagined. A few additional examples can be found in Malhotra's work [3] on a hypothetical management information system. These examples show clearly that question answerers cannot operate on a one-database-query-per-sentence basis.

A third result was discovered while testing the Automatic Advisor . It pertains to pragmatic relationships between elements of a database domain.

One of the data domains of the Automatic Advisor is that of the topics that are covered by the engineering courses in the database. The topics covered in the lower level courses were general, those in the upper level courses were more specific. The Automatic Advisor included no pragmatic relationships among the elements of the topics domain. Consequently, when asked for the courses that dealt with computers, it returned one course, the only one which explicitly mentioned "computers" as a topic. It failed to return about ten higher level hardware and software courses and a host of others that made extensive use of computers.

The problem of misleading answers due to missing pragmatic relationships is not peculiar to the Automatic Advisor, but is common to "portable" natural language processors. Harris states that "(the portability of ROBOT] is largely possible because of the way ROBOT makes use of the database as an existing semantic structure" [1]. Indeed, ROBOT, the Automatic Advisor, and other systems are evidence that much semantic and pragmatic information does lie in the structure of the database, but the tests have illustrated the limitations of relying solely on the database for this information. This suggests that conceptual coverage should include the pragmatic relationships between data elements even though these relationships may not be expressed in the database.

4.0 Conclusion

There is a clear need for natural language processors to be evaluated, The measures and measurement techniques described in this paper suggest that meaningful testing can be performed and that its results can promote the engineering of better systems.

The findings discussed above suggest that in addition to the study of the elements of linguistic coverage, attention should be focused on conceptual coverage. Without extending conceptual coverage beyond the limits of the database contents, a natural language question answerer can do little more than a formal query language. One mainstay argument in favor of natural language processors over formal query languages is that the use of natural language processors can be learned quickly and recalled easily after a period of disuse. These claims have not been substantiated, however.

The LADDER system [ 6] was evaluated [ 4) as a query language, and users were able to use the system with some facility after "moderate training (approximately 1.5 hours) in LADDER syntax and vocabulary" [ 4] (they were selected to be familiar with the activities described by the database), The LADDER users were trained because, "LADDER is sufficiently demanding in its syntax and lexicon so that if n.o training were provided, the outcome could be predicted with certainty -- namely, the technology would be severely limited as a tool for accessing a Naval command and control database." But tests of formal query languages have shown similar performance after familiarization (5].

It seems clear that the role that natural language question answerers should play is in providing the user with more complete conceptual coverage than he can have using a formal query language. The user has gained little if he avoids the rigors of learning a formal query language only to be subjected to the rigors of learning the details of the structure of the database. If he must know the minutia of either, he wi 11 not be able to use the system effectively on an occasional basis.