Strolling Down The Garden Path: Error Prone Tasks in Expert Problem Solving

1. I NTRODUCTION

Clas ses of problem solving ta s ks exist in which errors regularly occur, even among (human) expe rt problem solvers. The usually desirable s trategy of beginning a task with fairly global concepts and then working towards more specific levels of de tail (1] frequently fails when the st r ucture of the problem is such that the initial choi ce of global representation either does not apply or else fails to emphasize the detail of knowledge required for a successful solution. Some types of problem structures have the addi­ tional property that the dominant global strategy they elicit appears to be ·correct when, in fact, it is not . Problems of this type can be refered to as "garden path problems" since they induce pe r f ormance that has the false appearance of suc­

ces s .

If garden path problems consisted merely of puzzles or tasks in which tricks or caveats had to be discovered in order for a correct solution to be obtained, they might be interesting curiosi­ ties , but the attention given to them by the sc i entifi c community would be small. What makes these problems interesting, however, is the fact t ha t they may exist in a wide variety of knowledge domains ( eg. (2 ; ). Thus, they are of relevance to the s t ud y of both cognitive science and expert syst ems .

This work was supported in part by the National

Science Foundation under Grant SED79-13036.

the pressure in the separated state is less. Recovery from error in either case consists of choosing an alternative problem representation, a more detailed line of reasoning, or both. When experts working on problems of this type see errors in their thinking, they not only recover and adopt the correct approach to the immediate problem, but they also frequently generalize this recovery both backwards and forwards. Not only is the previous error recognized, but additional problems of the same type are solved correctly. Novices, by contrast, do not generalize either backward or forward and will frequently abandon a correct approach when given a new problem.

3. A FRAMEWORK FOR MODELLING

Observations of human performance on error prone tasks provide important information for the development of formal models of problem solving. Not only can the limits of competency be investi­ gated, but it is often easier to probe for the underlying mechanisms when the problem solving process has been perturbed by applying it to situations likely to result in incorrect decisions [4j. Studying the differences in performance between experts and novices on the same class of tasks gives additional insight by allowing compar­ isons of problem solving using different knowledge bases. We have been investigating the performance of human subjects on a variety of error prone problem solving tasks in physics with the aim of developing a computational model of this process. Our preliminary results indicate that a hierarchi­ cal p~anning process [SJ is employed in solving these problems. Unlike previous work in this area, however, we find that there is no clear level of primitive actions at which the validity of a plan can be confirmed.

3.1. Hierarchical lines of reasoning

To solve all but the most trivial problems, individual actions must be organized in an effi­ cient way. For many task domains, expert (human) problem solvers often structure their knowledge into "lines of reasoning" partially or fully ordered sequences of conceptual actions which are useful for specific types of problems [6,7j. Sig­ nificant efficiencies arise because these sequences are predetermined and need not be rediscovered for each new variant of a problem. A general class of tasks will typically have many lines of reasoning associated with it. These lines of reasoning can be characterized by the categories of knowledge used and by the level of detail with which that knowledge· is represented. Problem solving occurs by identifying an adequate line of reasoning, mapping the problem specifica­ tion into the appropriate representation, and then executing the implied inference steps.

If problem solving knowledge is structured in terms of levels of detail, then it is often possi­ ble to further reduce the set of actions which must be searched in order to find a solution. Problems can be first solved at an abstract level of representation and this solution can then used to constrain the possiblilities at more detailed levels [Sj. The garden path errors in our example occur because many problems are approached using a hierarchy of lines of reasoning in which the more detailed units of knowledge are often never exam­ ined. Such an organization appears necessary for solving complex problems with limited computa­ tional resources. Efficiency is achieved by ter­ minating the search through the hierarchy whenever it appears that the (abstract) solution is likely to be correct. Thus, garden path errors represent examples of the failure of powerful heuristics that, for the most part, greatly extend the effec­ tiveness of a problem solving system.

A general class of tasks will typically have many lines of reasoning associated with it. These lines of reasoning can be characterized by the categories of knowledge used and by the level of detail with which that knowledge is represented. For example, two major lines of reasoning are sug­ gested by the MILKO problem. One is based upon pressure, forces, and cross sectional area and is formalized by P · F/A. The other is based upon pressure, density, gravity, and fluid height and is formalized by P · pgh. Either line of reason­ ing leads to failure when applied at the global level where concepts in the corresponding equa­ tions are instantiated in terms of the total fluid in the container. Either line of reasoning can lead to success when applied at a more detailed level where concepts are specified in terms of decomposed problem sta~es.

3.2. Control issues

The manner in which control is exercised within and among lines of reasoning is central to the modeling of expert problem solving behavior. The desirable breadth of search has been a con­ tinuing topic among the artificial intelligence community. Narrowly structured searches promise potential efficiency, but easily result in garden path errors. Broader search sacrifices "best­ case" efficiency for improvements in accuracy. Our studies of human problem solving suggest that breadth of search is a function of both problem type and level of expertise [Bj. Ill structured problems or those for which limited knowledge (expertise) is available will result in less effective heuristics for recognizing the appropri­ ate line of reasoning. This must be compensated for by control functions that lead to greater breadth in the search process and thus allow fo r the simultaneous consideration of more possiblil i ­

ties.

In human problem solving, differences appea r in the ways in which experts and novices implement lines of reasoning. If a novice abandons a par­ ticular line of reasoning and then returns to it later in the problem solving process, the line of reasoning is usually reinvestigated from the beginning. Both the space savings and time inef­ ficiencies of classical backtrack programming are present. Few intermediate results need be remem­ bered, but many sub-problems may· be re-solved several different times. Experts, on the other hand, seem to utilize some sort of coroutine structure. Lines of reasoning are reactivated essentially at the point at which they were suspended. This may be possible because more finely developed control heuristics result in con­ sideration of far fewer possibilities, allowing a small set of partial results to be preserved.

4. SUMMARY

Garden path problems have been identified and studied in task domains as diverse as physics and medicine [8;. Such problems may be a natural consequence of hierarchically organized lines of reasoning created to support expert problem solv­ ing. We hope that our analysis can eventually lead to improved frameworks for knowledge engineering efforts as well as more complete models of cognitive behavior in problem solving.

ACKNOWLEDGMENTS

Robert Sedlmeyer and Richard Keller provided valuable comments during the preparation of this paper.

BIBLIOGRAPHY

• [ 1 I J .H. Larkin, "Teaching problem sloving in physics: The psychological laboratory and the practical classroom," in Problem Solving in Education, D.T. Tuma and F. Reif, eds., Hillsdale, N.J.: Lawrence Erlbaum Associates, 1980.
• [2j R. Peierls, Surprises in Theoretical Physics, Princeton: Princeton University Press, 1979.
• [ 3 I H.A. Cohen, "The art of snaring dragons, .. HIT AI Memo 338, 1975.
• [4~ H.A. Simon, The Sciences of the Artificial, Cambridge: H~Press, 198~ ---
• [5i E.D. Sacerdoti, A Siructure for Plans and Behavior, New York : Elsevier~rth-Holland: 1977.
• [6: E.A. Feigenbaum, "The art of artificial intelligence: Themes and case studies of knowledge engineering," Proc. Fifth International Joint Conference-on-Artif1CTSI Intelligence,~
• [7I J.H. Larkin, J. McDermott, D.P. Simon, and H.A. Simon, "Expert and novice performance in 'solving physics problems," Science, vol. 208, pp. 1335-1342, 1980.
• [8I P.E. Johnson, A. Barreto, F. Hassebrock, J. Moller, M. Prietula, P. Feltovich, and D. Swanson, "Expertise and error in diagnostic reasoning," Cognitive Science, 1981 (in press).