A Case Study in Structured Knowledge Acquisition

A CASE STUDY IN STRUCTURED KNOWLEDGE ACQUISITION

Paul de areef & Joost Breuker

Department of Social Science Informatics University of Amsterdam, Herengr~ht 196 1 0 16 BS Amsterdam

aimed at refining (and If necessary rejecting) an interpretation model.

Crucial to the methodology is the use of thinking aloud data. They provide the most informative window to expertise in action. However, in knowledge engineering, these data are hardly ever used. They are assumed to be difficult to interpret (Welbank, 1984) and their use is only recommended as 11 check on the adequacy of a prototype. However, as psych!)logy of problem solving shows, the analysis of thinking aloud data is feasible when an Initial model of the task is used as an interpretative framework (Ericsson & Simon, 1984). In KADS, a classification of such models is available. The knowledge engineer selects one or more interpretetion models, describing the expert tasks at a global level. There are Interpretation models for specification, diagnosis, planning, design, etc. The advantage of interpretation models is that the knowledge engineer is equiped with a tool that is much closer to the verbal data than an Implementation formalism. There are more pr~tical advantages: repair and refinement of the model does not require throwino awey some prototype !J)Ile stuck into a med!le of ed hoc solutions in an inappropriete forme !ism.

II Knowl!!!Q! Acquisition

The KADS methodology prescribes a series of knowledge engineering tasks which can be classified into three types:

1. An analysis of the functions, the environment and the users of the expertise to arrive at a definition of the operational char~teristics of the prospective system. The functional analysis defines the modality of the expertise. A knowledge based system contains two types of tasks: problem-solving tasks representing the expertise and communication tasks. These communication tasks are by no means trivial; they form the interface between the operational environment and the expertise. Modality mey involve negotiating, explanation, ~hing, documentation, etc.
2. An analysis of the static domain knowledge, starting with the collection of a lexicon. ending with concepts structured In ( KLONE) concept hierarchies.
3. Analysis of expertise In ~lion, I.e. the wey problems are solved. This starts with a task-analysis: selection of one or more interpretation models that appear to represent the structure of the problem solving process. By metching the verbel dele from interviews and in particular thinking aloud protocols, this initial model gets refined and modified Into a detailed structure of knowledge objects, knowledge sources and stretegies; much in the same wey as Bennet's ( 1984) conceptual structures. The final conceptual structure of expert reasoning represents the basic architecture of the prospective system. In the conceptual structure

ABSTRACT

Building an expert system usuelly comprises en entengled mixture of knowledge ~K:Q~Jisition and implementation efforts. An emerging methcmlogy based on cognitive psychology and software development guides end supports knowledge ~K:Quisition while implementation is deferred. This allows for a more deliberate choice of architecture. This paper presents a case study to test the methodology.

I INTROPUCTIQN

A major problem in the construction of expert systems is the method for knowledge ~K:QUisition. This mey be one of the reasons that despite a widespread need there is little advance in building intelligent consultants for statistical problems. There are two distinct methods. The first one is the methodology of rapid prototyping in which knowledge ~uisition and implementation are mixed. The knowledge engineer uses interview data from humen experts and immediately starts building a prototype in an implementation formalism. How to make decisions regarding implementation and architecture remeins unclear (cf. Heyes-Roth et.al. 1983). The second methodology, which we call structured knowledge ~uisltion, is outlined by Wielinga & Breuker ( 1984 ). This case study is meant as a test of this KADS methodology (Knowledge Acquisition Documentation and Structuring). The methodology is implemented in a knowledge ~uisition support system, KADS, written in Pro log, using a Pro log KLONE implementation as a structuring device.

A major cher~teristic of the methodology is a separation between knowledge ~ulsltlon and Implementation. The task for the knowledge engineer is to bridge the gap between the verbal data from experts and the ~tual implementation of a system. The methodology provides e theoretically founded step in between. It guides the knowledge engineer in the mapping of verbal data onto an Intermediate level provided by an interpretation model, which is an implementation independent description of the domain knowledge on an epjstemol~jcal level (Br~hman 1979, Clancey 1983). It consists of a typology of basic elements and structuring relationships for a certain class of problem solving tasks.

The basic elements ere objects, knowledge sources, models end strategies. On the implementation level, knowledge sources can be algorithms or sets of production rules. On the epistemological level a knowledge source is a piece of knowledge that derives new information from existing data. It is equivalent to an elementary subtask, which cannot be decomposed further. A knowledge source thet occurs in almost any problem solving task is for instance the classification of objects into categories. Knowledge ~uisition consists of repeated cycles of elicitation end enalysis of verbal date the static knowledge and the ections performed on them become integrated.

In this section we discuss these l<nowleO;le engineering tasks within the domain of stetisticel enalysis of experiments. From !<now on "tasl<" means tasl< for a statistical consultant. The deta used in this case stl.Kty' consisted of textbooks on statistics and. tw~lve thlnl< l.ng aloud protocols, obtained from four experts wor1<1ng m the soc1al science department at our university. One of us is an intermediate expert himself.

The funct1onal analysis alms at Identification of the objects, egents end functions involved in the expert tasl<. An inform81 description of the functional analysis is the following: A statistics expert is consulted by a researcher who has the Intention to investigate empirical relations among variables. The specification of these intentions is celled a CONCEPTUAL MODEL. It contains conceptuel veriebles end reseerch questions ebout conceptuol relations among them. This CONCEPTUAL MODEL is transformed into a RESEARCH PLAN for the collection of empirical DATA. This plan containins steps lil<e SAMPLING, MEASUREMENT, etc. The transformation is usually done by the researcher, but this may lead to BUOS in the research plan. Abstrect properties of the research plan -the underlying DESIGN- determine which research questions can be investigated and by which ANALYSIS MODEL the d6t6 should be analyzed. A BUG is e property of the reseerch plan thet prohibits 8 clear end correct answer to a specific research question. The expert proouces an ADVICE which consists of an ANALYSIS METHOD and identified BUGS in the RESEARCH PLAN.

The domain lexicon Is collected from verbal deta and textbooks. It provides a vocabulary to communicate with experts, and to identify the domain specific concepts. These concepts need to be structured in hierorchies. One opproach is to use experts or textbooks to identify the general concepts at the top. However, such an approach is naive. because these structures rather renect the support knowledge thet is used for a theoretical justification of statistical designs than the way the expert uses concepts while performing his tasl<. Initial structuring is provided by the I< now ledge objects specified by the functional analysis end knowledge objects which are the lnterfeces between knowledge sources in the conceptuel structure of the reiiSOiling. During the enelysis mllflY new concepts were added, because statistics textbooks lecl< concepts

relevant to application.

In the task analysis an initial template interpretation model was selected, derived from a problem solving system in the main of thermodinamics ( Jansweijer et. al , 1982). These mains have in common that an informal problem is transformed into a formal structure to which forme! principles can be epplied. The templete model describes a first st8IJI ORIENTATION in which the process SKETCH obtains an overview of the Informal problem situation which is tronsformed by SCHEMATIZE into 8 forme! problem statement. The second stage solves the problem and is followed by an evo1U6tlon stage.

The thinking aloud protocols were especially useful for identification and refinement of the l<nowleO;je sources in the successive interpretation models.

Ill CONCEPTUAL STRUCTURE OF TA5KSANO SVSTEM DESIGN

An overview of the interpretation model for the ORIENTATION stage is presented in fig. I (more deteils cen be found in the Knowledge Acquisition Document (De Greef, 1984)).

Fig. I: tesl<s in the ORIENTATION stage. (OVals represent types of I< now ledge sources, rectangles represent types of I< now ledge objects.)

The control of the problem solving process is guided bye plan that can be expressed as a tree. A general plan for execution of these tasks is shown In table 1.

Table 1: Tasks in the orientation staoe of statistical consultation.

ORIENTATION

SKETCH

SPECIFY CONCEPTUAL MODEL SPECIFY OPERATIONAL DEFINITIONS UPDATE RESEARCH PLAN

SCHEMATISE

CLASSIFY OPERATIONAL VARIABLES ABSTRACT THE DESIGN-STRUCTURE

A knowledge source may modify the plan. For instance, SPECIFY OPERATIONAL DEFINITION may run into trouble, because the client cannot answer a question or because the current information leaves an ambiguity about the RESEARCH PLAN. Then, before flnishinQ and returning control, it may create a new subplan. SPECIFY RESEARCH PLAN. SUch change of stretew wes frequently observed with human experts. Another instance: CLASSIFY OPERATIONAL VARIABLES may conclude that specific properties of certein OPERATIONAL VARIABLES prohibit the investigetion of e specific RELATION in the CONCEPTUAL MODEL. Then a knowledge source of the DESIGN-BUG type Is Inserted as a new subplan. DESIGN- BUG will communicate the conclusion to the user and change or aUQment the CONCEPTUAL MODEL. Here human experts often try to improve the RESEARCH PLAN. Since this is plen is usuolly executed and done this is of little precticel value and need not be included in the ~tern . The final product of the orientation stage is a forme! problem stetement. The next stege selects e model thet defines which relations can be investigated and which statistical models should be used. There exist 8lgorlthms to do the ectU!Il assembly of statistical analysis models, provided that certain assumptions about the dependent variables can be made. If not so, a decision tree is sufficient to find an assumption free analysis technique, if there exists en app 1iC8b le one.

A most critical task for a consultation system is obtaining the correct problem statement from the client. Many misinterpretations may occur. as is shown in a number of protocols. A misinterpretation is usualy discovered by the client after heering 8 solution from the expert. This causes a very complex process of debugging the problem statement (CONCEPTUAL MODEL and/or RESEARCH PLAN) and Ojltation of the solution ( the ADVICE). The system can be m~ less complex if it lets the user evaluate the correctness of the problem statement before attempting to solve the problem. For evaluation by the user and subsequent refinement and debugging of the problem statement we propose the system displays graphical structures which can be edited by the user. For the CONCEPTUAL MODEL, a labeled graph can be used. For the RESEARCH PLAN it can use flow diagrams as proposed by 0' Keefe ( 1981 ).

The system desiQn discussed so fer. is suitable for users with little but sufficient experience in statistics. A user with insufficient knowleO]e will not be able to answer all questions. More sophisticated users may be bored or irritated, not being able to use any shortcuts. Although a system like this will be useful as a front end to a set of statistical software packages, there is still something to be desired: adaptation to users with different levels of statistical knowleO]e, explanation of bugs in the reseerch plan, use of the system as a helper in selecting a design and mal<ing a research plan (instead of being only a critic), etc. These properties h8ve nothing to oo with the statistical problem-solver but with the mod81i1¥ of coosultaocy. It is desirable that a consultant has qualities of en Intelligent Tutoring System end is also a capable research designer and planner.

IV IMPLEMENTATION

We have implemented a small prototype. It can solve correlational end experimental problems following the plan outlined in table 1 and two other plans. The concepts are part of a KLONE structured inheritance network ( Brachmen 1979), implemented in ProlOJ. The interpreter language which can be translated to ProlQJ is borrowed from the PDP problem solver (Jsnsweijer et. al. 1982). After six men months of knowleO]e acquisition , it tool< four weeks each for two computer science sturents to implement a shell for the prototype using the PDP tools and the knowleO]e acquisition document in De Greef ( 1984). The prototype 008s not yet contain user editable graphical displays.

V CONCLUSIONS

The case study shows that the methaoology is viable and that a separation between analysis and implementation is possible. The result of anelysis is a knowleO]e acquisition document which provides a sufficient basis to enable two students unfamiliar with the statistical oomain to implement a prototype in short time.

The methaoology is en alternative to the methoo of rapid prototyping, in which a knowleO]e engineer very quickly mal<es architectural decisions and builds a prototype. The prototype is shown to the expert who is expected to provide useful criticism. Then the prototype is chenged or a new system is built. This cese stu!t{ used a methaoology in which major commitments to architecture and implementation efforts are postponed until it is clear whet hes to be implemented.

A conceptual structure of a task can be evaluated by an expert in much the same way as a prototype. The difference is that the evaluation is focussed on the the problem solving process; not on the performence of e system. Moreover, chenging e conceptuel structure takes far less effort, than debuooing a prototype system. We have experienced that this methcxl of obtaining feed back on the interpretation of expertise in action is less time consuming and probably as effective as evaluating a prototype, whose inner workings may eppear rather obscure to the expert.

ACKNOWLEDGMENTS

We want to thank the OO!nain experts for sharing experience and thougts, Kees Buisman and Dick v.d. Vlugt for their work on the prototype, end Robert de Hoog and Bob Wielinga for valuable advice.