{"type":"ScholarlyArticle","title":"An Achievement Test for Knowledge-Based Systems: QUEM","authors":[{"name":"Caroline C. Hayes","slug":"caroline-c-hayes","given":"Caroline C.","family":"Hayes","affiliation":""},{"name":"Michael I. Parzen","slug":"michael-i-parzen","given":"Michael I.","family":"Parzen","affiliation":""}],"abstract":"This paper describes QUEM, a method for assessing the skill level of a knowledge-based system based on the quality of the solutions it produces. QUEM is demonstrated by using it to assess the performance of a particular knowledge-based system, P 3 . QUEM can be viewed as an achievement or job placement test given to knowledge-based systems to help system designers determine how the system should be used , and in what capacity by what level of users . In general, it is difficult to find useful metrics for assessing a system's overall performance . Most literature on evaluation deals with validation, verification and testing in which the primary concern is the correctness and consistency in the databases and rulebases . However, these properties alone may not be sufficient to determine how well a system performs its task. QUEM allows software developers to assess their system's performance by constructing a skill function based on human performance data that relates experience and solution quality. QUEM can be used to gauge the experience level of an individual system, compare two systems , or compare a system to its intended users . This represents an important advance in quantitative measures of over-all system performance that can be applied to a broad range of systems .","doi":null,"url":"https://ijcai.org/proceedings/1997/43","pdf":"https://ijcai.org/proceedings/1997/0043.pdf","markdown":"https://ijcai.org/proceedings/1997/43.md","bibtex":"https://ijcai.org/proceedings/1997/bibtex/43","volume":{"slug":"1997","title":"Proceedings of the Fifteenth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 1997","url":"https://ijcai.org/proceedings/1997/","year":1997,"location":"Nagoya, Japan","dates":"","editors":"Martha E. Pollack","isbn":"","publisher":"International Joint Conferences on Artificial Intelligence","doi_prefix":""},"section":"Cognitive Modeling","track":"Cognitive Modeling","number":43,"pages":{"first":288,"last":292},"keywords":[],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}