Semi-Automated Analysis of Protocols From Novices and Experts Solving Physics Problems

1. Introduction

The goal of our project is to investigate differences between novice and expert problem solving m semantically rich domains in order to explicate what is learned by doing. A basic assumption is that the behaviour of a human problem solver is determined by his knowledge (or lack of knowledge) about the problem domain. Thus the main question to adress is what changes take place in the knowledge base of a novice becoming an expert.

The domain chosen is that of elementary thermodynamics. The semantic richness of this domain does not allow the observation of subjects during the entire skill acquisition process from novice toward expert level. The methodology adopted instead is to observe subjects at two different levels of performance: novice level and intermediate level, and to simulate this behav.iour with a computational model for each of these levels.

Since the problem solving behaviour of human subjects, novices in particular, is idiosyncratic and unpredictable we do not aim at predicting all behaviour in detail, but adopt the strategy of s.emi-automatic simulation (Bhaskar and Simon, 1977): whenever the model fails to simulate the behaviour of the subject, it can be put back on the right track by a human coder. In this way protocols of subjects at different levels of expertise can be analyzed in terms of the computational model and annotated with discrepancies between model and subject.

· Netherlands Organization for the Advancement of Pure Research (ZWO).

2. Description of the system

The PDP system consist of three components. The oroblem solving component (PERFORM) is implemented using a locally developed (Tnterlisp) implementation of KL-ONE (Brachman, l978). In contrast with Bhaskar and Simon (1977), the PERFORM component actually solves the problem. The KL-ONE network contains both the declarative knowledge about the domain, and procedural knowledge represented as concepts with attached demon-like Lisp-procedures. Procedural knowledqe is activated by a scheduler using a multi-level agenda similar to KRL. All operations on the KL-ONE network (e.g. filling slots, individuating concepts), generate "events" which are sent to a second component of the system: ANALYZE. ANALYZE interprets the event with a production system and decides whether the event will be translated into natural language utterances. These utterances are presented to a human coder for comparison with a subject 's protocol. If there is a match between an event and part of the protocol, the line numbers of that part are typed in by the coder and a signal is sent to PERFORM to continue. If the behaviour of the program and the subject do not match, a third component of PDP is activated: DIAGNOSE.

The DIAGNOSE component is equipped with rules representing knowledge about deviations from the normal line of behaviour of the model (e.g. errors and impasses). In interaction with the coder DIAGNOSE can identify the problem and redirect the behaviour of the model (e.g. by changing knowledge structures in the working memory or by reactivating certain procedures).

The Output of the model is an analysis of the protocol in terms of the procedural and declarative knowledge used by the mode!. Discrepancies between model and protocol are also indicated.

3. Intennediate level model

Aa a starting point for the intermediate level model, an instructional method developed for an undergraduate level physics cours e was used, which prescribes the ac tions and methods needed to solve problems in thermodynamics (Mettes et al., 1981). The procedural knowledge of the model consis ts of three main procedures: ORIENT ATE, SOLVE and EVALUATE. The task of ORIENT ATE is to read the problem and to anal yze the problem in terms of concepts relevant to the physics domain. The output of ORIENTATE is a transformed representation of the problem suitable for use in the solving phase.

SOLVE implements a backwards search for a sequence of equations which will solve the problem. When such a sequence is found, numeric values are filled in the equations and the answer is computed.

EVALUATE currently only implemented in a rudimentary form- checks the consistency and plausibility of the answer against qualitative expectations etc. More details of the intermediate level model are given in Jansweijer et al. (1982).

With the intermediate level version of PDP we analysed 17 protocols of a second year physics student. More than 80% of the protocol statements could be matched with corresponding activities of the model. In general, the model and the behaviour of the subject were in concordance with respect to the order in wich different procedures were ac tivated. The table below shows the relative number of activities performed in the three main procedures of the model, compared to the relative number of ac tivities actually identified in the protocols.

The data indicate that the relative amount of processing performed in different phases of the problem solving proces, is accurately predicted. This is the more remarkable since the model puts much more emphasis on the orientation phase than most previous models of physics problem solving. The MECHO program (Bundy et al. 1979; Luger, 1981) for instance starts to select a physical principle and makes most of the necessary inferences and transformations when particular pieces of information are needed. The ABLE model (Larkin 1980) also concentrates to a large extent on the ways in which principles are selected. Our model is more similar to the ISAAC program (Novak, 1977) in its emphasis on transformation of the problem representation before actual principles are selected.

As can be seen from the above table the model performs almost twice as many activities as are present in the protocols. One reason for this is that the subject does not verbalize all of his activities in the same detail as PDP does. Another, more important, reason for this phenomenon is that, during the course of the experimental session, the subject learns from his own problem solving efforts: the type of problem is recognized, certain operations become automatic, analysis of problem features becomes more goal oriented, seemingly guided by problem specific schemata. This learning results in a smaller number of (verbalized) activities, in particular during the orientation process. The model however is not equipped with any learning capabilities. Quantitativily this can be illustrated by comparing the results for a ·series of problems occuring early in the sessions and an isomorphic series presented at a later stage. In the late problems we saw only half as much ORIENTATION activity as in the early isomorphic problems. For SOLVE the difference is less: 85 % of the activiti es remain present.

Some activities of the subject could not be coded in terms of the model. Thes e a re: recognition of problems as similar to previously solved ones and checkinq of dimensions.

4. Novice level model

Analysis of novice protocols, with the intermediate version of PDP showed us that novices behave quite differently: the order in which novices perform the various activities deviates strongly from the model. In particular the Orientation process is only present in a rudimentary form, resulting in an incomplete analysis of the problem. This causes problems during later stages of the problem solving proces. The subject gets into "impasses" and needs to perform problem solving activities to overcome the impasse.

On the basis of these general observations a first model at novice level was constructed. I,t differs from the model at intermediate level in two ways: First, a number of procedures are deleted from the intermediate model, in particular those related to transformation of the problem representation. Second, a number of repair procedures were added to the DIAGNOSE component of the model (c. f. Brown & vanLehn, 1980). When the problem solving proces gets into an impasse, DIAGNOSE tries to classify the cause of the impasse and suggests one or more repairs. The human coder select» applicable repair by inspect ing the novice protoc·

For example, whe principle is selected for which the preconditions are nut satisfied, the system gets into an impasse. Two oossible repairs are then suggested: "reject principle" and "relaxation". When the coder selects the first repair, the principle is rejected and another principle is selected. When the coder selects the second repair, continuation of the proces with the selected and possibly wrong principle is forced. Another typical example of an impasse is when the system ends up with a filled equation in which there is no unknown left to be computed. In such a case the repairs "start anew" and "reject principle" are suggested.

A summarized example of an output-trace of PDP for a novice protocol is shown below. The uppercase weirds denote the procedures of the mode!.

RESEARCH-DIRECTOR ORIENTATE READ-PROBLEM ABSTRACT -CUES SOLVE SELECT -PRINCIPLE IMPASSE (precondition not satisfied) REPAIR RELAXATION COMPUTE FILLING-EQUATION TRANSFORM-UNITS IMPASSE (transform a~ · REPAIR IGNORE_ CALCULATE EVALUATE ,f units failed) .:J

The first activity of the model is to read the problem and abstract relevant cues from the problem text. On the basis of these cues, a principle is selected. Because the precondition of this principle is not met, the system gets into an impasse. Since the subject ignores this precondition the impasse is repaired by relaxation of the precondition. The values of the variables are now filled in the equation. The procedure which transforms units fails, due to lack of knowledge and the resulting impasse is repaired by ignoring the units altogether. Finally the asked is computed and the solution is evaluated. The figure above represents problem-solving behaviour typical for a novice, in which the orientation process is only present in a rudimentary form.

5. Discussion

The intermediate level model fits the behaviour of experienced subjects quite well, indicating that subjects with good knowledge of physics and experience in solving this type of problem, are able to work systematically and do perfm:m an extensive analysis of the oroblem before selecting equations.

The model for novice behaviour, based on the assumption that lack of procedural and declarative knowledge causes impasses which are resolved by local repairs produces behaviour which is typical for novice protocols. A substantial number of protocols have been analyzed using this model. Although the model accounts for a large percentage of protocol utterances, a number of repairs had to be introduced in a rather ad hoc way. Also, non-local repairs were observed in the protocols (e.g. reactivation of the orientation proces when the solve proces gets stuck), which are not easy to model in this way. In contrast with Brown & vanLehn (1980), we have not been able to identify and implement a parsimoneous repair strategy.

Currently a new version of the novice model is under construction, which is controlled by goals embodied in a global plan. In this system the DIAGNOSE component will be able to generate subgoals, whenever an impasse occurs. In this way, we expect to gain flexibility in control over repairs.