---
title: "An Achievement Test for Knowledge-Based Systems: QUEM"
authors:
  - "Caroline C. Hayes"
  - "Michael I. Parzen"
doi: ""
url: "https://ijcai.org/proceedings/1997/43"
pdf: "https://ijcai.org/proceedings/1997/0043.pdf"
venue: "Proceedings of the Fifteenth International Joint Conference on Artificial Intelligence"
year: 1997
pages: "288-292"
track: "Cognitive Modeling"
keywords:
---

# An Achievement Test for Knowledge-Based Systems: QUEM

Caroline C. Hayes, Michael I. Parzen

*Proceedings of the Fifteenth International Joint Conference on Artificial Intelligence*, pages 288-292.

## Abstract

This paper describes QUEM, a method for assessing the skill level of a knowledge-based system based on the quality of the solutions it produces. QUEM is demonstrated by using it to assess the performance of a particular knowledge-based system, P 3 . QUEM can be viewed as an achievement or job placement test given to knowledge-based systems to help system designers determine how the system should be used , and in what capacity by what level of users . In general, it is difficult to find useful metrics for assessing a system's overall performance . Most literature on evaluation deals with validation, verification and testing in which the primary concern is the correctness and consistency in the databases and rulebases . However, these properties alone may not be sufficient to determine how well a system performs its task. QUEM allows software developers to assess their system's performance by constructing a skill function based on human performance data that relates experience and solution quality. QUEM can be used to gauge the experience level of an individual system, compare two systems , or compare a system to its intended users . This represents an important advance in quantitative measures of over-all system performance that can be applied to a broad range of systems .