An Achievement Test for Knowledge-Based Systems: QUEM

Abstract

This paper describes QUEM, a method for assessing the skill level of a knowledge-based system based on the quality of the solutions it produces. QUEM is demonstrated by using it to assess the performance of a particular knowledge-based system, P 3 . QUEM can be viewed as an achievement or job placement test given to knowledge-based systems to help system designers determine how the system should be used , and in what capacity by what level of users . In general, it is difficult to find useful metrics for assessing a system's overall performance . Most literature on evaluation deals with validation, verification and testing in which the primary concern is the correctness and consistency in the databases and rulebases . However, these properties alone may not be sufficient to determine how well a system performs its task. QUEM allows software developers to assess their system's performance by constructing a skill function based on human performance data that relates experience and solution quality. QUEM can be used to gauge the experience level of an individual system, compare two systems , or compare a system to its intended users . This represents an important advance in quantitative measures of over-all system performance that can be applied to a broad range of systems .