Progress On A Computer Based Consultant

Introduction

One of the increasingly prominent trends in computer science research has been the emphasis on incorporating specialized bodies of knowledge in computer programs and making this knowledge conve­ niently available to users who are not computer ex­ perts. Such programs- -which we might call computer based consultants--can be viewed as stemming from the confluence of two lines of research. One line of research has centered on formulating and en­ coding a great deal of knowledge about a chosen problem domain in order to produce a program whose performance rivals expert humans. Often cited ex­ amples of this research include programs that ana­ lyze chemical structure,1,2 perform symbolic inte­ gration,a or play board gaines well.4, s, s

The second line of research has focussed on methods for constructing a program that can carry on a dialog with a user. Important contributions to this research have come from work in computer aided instruction, and from work in understanding typed and spoken natural language. Representative examples of this work include programs to carry out a "mixed initiative" tutorial dialog,~ a to engage in a dialog about a toy block world 9 and to under­ stand spoken English sentences about such diverse top i cs as plumbing,l O news stories, ll moon rocks,l 2 or submarines.l3

The work reported herein was sponsored by the Advanced Rese.arch Projects Agency of the Department of Defense under Contract DAHC04-72-C-0008 with the U.S. Army Research Office.

In this paper we describe initial progress on another computer based consultant. This new con­ sultant is aimed at helping an inexperienced me­ chanic work with mechanical equipment. Before de­ scribing the functional components of the system, let us first consider some of the characteristics of the problem domain.

The Problem Domain

Imagine a mechanic (whom we will assume to be relatively inexperienced) working on a piece of equipment in a "work station" like the one sketched in Figure 1. He is typically responsible for a variety of jobs, such as troubleshooting, repairing, or modifying equipment. In order to do these jobs, he needs certain kinds of specialized knowledge; he must know about the use of various tools, about principles of troubleshooting, and about principles of assembly and disassembly, and he must also know a certain amount of detail about the construction and operation of the specific equipment on hand.

A traditional way of conveying this knowledge to a mechanic has been through the use of manuals. A more nearly ideal, though usually impractical, way woutd be to make an expert mechanic continu­ ously available as a consultant. The expert could identify various components, answer specific ques­ tions about equipment details, suggest trouble­ shooting sequences, hypothesize causes of failure, warn of hazards, and so forth.

In order to explore what would be involved in replacing the human expert by a computer based ex­ pert, we recorded a number of dialogs between ex­ pert human mechanics and novice mechanics . The di­ alogs concern the air compressor shown in Figure 2. (We shall use this compressor throughout the paper for illustrative purposes.) Two excerpts from these dialogs are presented below. At the time the di alogs were recorded, the expert and novice were i n different rooms and the expert viewed the scene only by means of still pictures taken through a

television camera. (We did this to simulate the limited visual information available to a computer based expert.)

The first excerpt concerns the subtask of in­ stalling a pump pulley on the pump.

Excerpt 1

Expert: Novice : Expert : Novice : Expert : Novice : Expert : Novice: Expert : Novice : Expert : The pump pulley should be next. Yes ... uh, does the side of the pump pulley with the holes face away from the pump or towards it? Away from the pump. All right. Did you insert the key--that is, the half-moon shaped piece? Yes, I did. Be sure you check the alignment of the two pulleys before you tighten the set-screws. Yes, I'm just now fiddling with that. OK. Tightening the Allen screw now. OK, thank you.

This fragment illustrates several important abilities of consultants that contrast sharply with a static information source like a manual. First, notice that a question from the novice is answered directly and in his terms. There is no need for him to search through a mass of information, or to convert information from an abstract or "standard" form into a directly usable form. Notice that the expert is checking on progress by offering warnings and reminders about critical steps. This has the function not only of minimizing errors, but also of allowing the expert to keep track of the progress of the work. The latter function is the basis for the expert's ability to present relevant advice, and to present it in a context that is familiar to the novice.

The second dialog expert concerns the same subtask, but was carried out with different partic­ i pants. It offers an interesting comparison of the different demands imposed by different skill l evels :

Excerpt 2

Expert: Install the pulley on the shaft.

Novice: What is the first thing to do in in­ stalling the pulley?

Expert : Rotate the shaft so that the slot

(keyway) is on the top.

Novice : OK ... now what?

Expert: Place the key in the slot .

Novice: Flat side upward? Expert: Yes.

This short fragment dramatically illustrates the ability of the expert to descend into detailed instructions in order to help a very naive user. This novice needs much more help than the first one did, a situation foreshadowed by his initial question about a relatively simple operation.

The short dialog excerpts exemplify some of the abilities that a consultant needs in order to be helpful to the novice mechanic. Both intro­ spection and protocol experiments point out a num­ ber of other required abilities, among which are the ability to provide advice about trouble­ shooting; to describe the use of tools; to de­ scribe the appearance of tools (or to be able to point them out}; and, of course, the ability to use language .

The abilities required of a mechanic's con­ sultant impose a number of technical requirements on a computer system designed to fill that role. It is worth mentioning just a few of these re­ quirements to illustrate the research problems we are addressing.

The set of problems that is perhaps the most characteristic of our project centers on providing advice about a task at any of several levels of detail, and of interacting with the novice as he uses this advice. In particular, mult i level plans must be created and represented, the novice must be modeled in order to determine the level of de­ tail he needs, his performance must be monitored as he executes the task, and internal models must be maintained to reflect the current state of the task environment. This, in turn, imposes a need for semantic representations that can accumulate a structured discourse history that evolves as the task proceeds, and that can provide linguistic contexts and clues to the novice's competence. Also, because so much information is communicated visually, we must be prepared to use vision to answer questions from the novice; but, because the world of machinery is exceedingly complicated vi­ sually, we must exploit geometric models and se­ mantic constrains extensively if we expect to be able to answer a reasonable range of "visual ques ­ tions . "

It is interesting to note that most of t he foregoing technical requirements are not peculiar to a mechanic's consultant; they are l i kely to un­ derlie any computer based consultant system. In the next sections, we will outline the progress we have made on these research problems.

Functional Components

Let us begin by considering the system orga­ nization sketched in Figure 3. The consultant

system interfaces with the physical domain of the work station through several devices. A headset enables the novice mechanic to talk to the system and to receive spoken replies; it represents a natural language component of the system. A vi­ sual component is represented by a color televi­ sion camera and a laser rangefinder. The laser rangefinder is a mechanically scanned instrument, developed for this project, that generates an ar­ ray of range values supplementing the color tele­ vision picture. The range array and picture ar­ rays (one for each primary color) can be placed in registration, providing a multisensory image that specifies the color of, and distance to, each point in the scene. The rangefinder can also be operated as a visual pointer, so that the system can answer questions like "show me the pressure switch" by pointing at it--that is, by illurili­ nating the pressure switch with the laser beam.

The raw sensory data provided by the trans­ ducers are translated into internal representa­ tions by the natural language and visual func­ tional components of the system. These internal representations trigger subsequent action. For ex­ ample, a question about an assembly step might be answered either by referring to an assembly se­ quence already stored in a model, or by using the planning capability to compose a sequency if the model does not contain an appropriate one. Simi­ larly, a question about the location of a part might be answered either by referring to a geomet­ ric model or by locating the part using new vi­ sual data.. Of course, the natural language and visual components themselves need various kinds of model information in order to translate raw data into internal representations. For example, i n order to understand a given sentence, it is necessary to access a discourse model in order to establish the referents for pronouns or deter­

mined nouns.

With th-is system overview as a background, we can review the current stock of ideas and programs for each of the functional components.

Planning for Assembly

The system component that has received the most attention to date has been the planner and associated model, for composing assembly and dis­ assembly sequences. We have emphasized this be­ cause assembly and disassembly are subtasks of virtually all typical work station tasks. For ex­ ample, many troubleshooting jobs and almost all repair jobs require some amount of disassembly and reassembly of the machine.

Let us use the task of assembling the air compressor to illustrate in a simplified way how assembly plans are produced.* Three different types of knowledge are used : a model of the spe­ cific air compressor, a procedural model that en­ codes more general information about how parts are fastened together (i.e., assembled), and a planner that has abstract knowledge about how plans can be represented and about how the steps of a plan can

interact.

The compressor assembly model is essentially a graph whose nodes correspond to the parts of the compressor (the motor, pump, pulleys, and so on), and ~ose arcs correspond to the mechanical con­ nection between parts. A considerable amount of information is usually associated with each arc. For example, the arc representing the connection between a pulley and its shaft may include infor­ mation about the set screws and key. (The key prevents relative rotation between pulley and shaft.) Similarly, the arc connecting the belt cover and its suppo.rt may contain information about the number and size of the sheet metal screws. This equipment-specific model also con­ tains certain auxiliary information peculiar to the compressor, like the fact that the pump cannot be installed if its pulley is already on the pump

shaft.

Each generic type of connection has associ­ ated with it a set of procedures that contain in­ structions about how that connection is physically accomplished. For example, a procedure associated with installing a pulley on a keyed shaft might include specific instructions about inserting the key and tightening the set screws. Note that this procedure is independent of any specific piece of equipment; it offers generally useful knowledge about how a certain job in the domain of mechani­ cal equipment is done, and it would be invoked whenever that job was necessary. In addition to the specific instructions, procedures of this sort contain calls to other procedures that elaborate in more de~ail how the given job is done. In our pulley and shaft example, we might want to call more detailed procedures for, say, describing how to align pulley and shaft or for dealing with rusty parts. This hierarchical structuring of pro­ cedural knowledge forms the basis for producing plans that can be stated to a novice at any of sev­ eral levels of detail.

* Disassembly plans are essentially similar, and will not be discussed explicitly.

The procedural model of assembly operations allows the planner to generate instructions about how to connect two specific parts, but it does not select the order in which parts are to be con­ nected. This is the job of the general planning program. The planner adopts the view that if there are n connections to be made between pairs of parts, all connections are equally important and that there is no prior reason to prefer any partic­ ular order. That is, it initially assumes that all n assembly steps will be made in parallel--logi­ cally, as a conjunction. However, it then expands the steps in greater detail, and examines the pre­ conditions and effects of these steps to see if there is any interference among them. To continue our example, it would discover that the pump can be installed only if there is no pulley on its shaft. This would interfere with a different as­ sembly step; namely, installing the pump pulley on the pump shaft. The planner recognizes this poten­ tial conflict, and imposes an order so that the pump will be installed before its pulley is placed on its shaft. When all conflicts of this nature are resolved, the remaining steps can indeed be logically performed in any order.

This ability to recognize alternative or­ derings of steps has major implications for any computer based consultant: A human performing a task may well take the initiative on occasion and choose an ordering for certain steps, and it is im­ portant for the consultant to know whether this choice is valid. Equally, the availability of al­ ternative orderings affords an opportunity to ap­ peal to other ordering criteria like ease of physi­ cal operations.

A plan is represented as a procedural net, a fragment of which is shown in simplified form in Figure 4. Each node corresponds to an assembly step at some level of detail. The net represents a hierarchy of plans, all accomplishing the same task but stated at varying levels of detail. The ith row of the net represents one complete plan at the ith level of detail, and the dotted lines in­ dicate the expansion of a step into a .more de­ tailed subplan. Notice that the Level 2 plan splits into two parallel paths at A and merges to­ gether at B in order to represent the fact that the two subplans can be performed in either order.

Procedural nets have proven useful in several ways. Perhaps the most obvious is that it allows us to specify a plan to the novice mechanic at varying levels of detail. Typically, the .novice will understand some steps at a high level and need little or no additional elaboration, whereas he will be mystified at other steps and need to have them expanded into more complete instruc­ tions. By keeping track of an execution path through the net, we can link steps at the various levels of detail. The more general problem here is to learn how to monitor the mechanic's perfor­ mance as he executes a task. We would, for exam­ ple, expect the system to ask occasional questions of the novice (just as the human expe!t did in the dialog) in order to monitor his progress as he proceeds through the net. Thus far, our system is not that flexible and monitors progress by ad­ hering to a more limited dialog format.

In addition to the several uses of procedural nets at plan execution time, they are also used during planning to represent partially formed plans. This allows us to restart the planner to modify an existing plan during the course of ita execution, which in turn permits us to respond to information discovered as the assembly physically proceeds. Further discussion of procedural nets and arbitrary orderings of plan steps will be found in a forthcoming paper.l5

Before leaving the subject of assembly plan­ ning we should mention a second type of hierarchy which is distinct from the hierarchy of plan de­ tails that we have been discussing. This second hierarchy deals with levels of equipment, and is motivated by the fact that often the major parts of a mechanical device are themselves components that can be assembled and disassembled. For exam­ ple, the pump of the air compressor has a piston, crankshaft, and valves and is in fact similar in some ways to a simple one cylinder gasoline en­ gine. We thus expect that the general scheme for assembly planning described above would be repli­ cated hierarchically to deal with several levels of components of the equipment. Ideally, then, we hope eventually to be able to provide consultation at several levels of detail about any of several levels of equipment components. We have only re­ cently begun considering this second hierarchy, but it appears to entail a relatively straightfor­ ward extension of the ideas discussed already.

Troubleshooting is a key element of a mechan­ ic's job, and often represents the task requiring the highest levels of skill and experience. In spite of its obvious importance, we have given it relatively little attention thus far because of our decision to first reach a reasonable level of competence at assembly planning. Accordingly, we can offer here only tentative remarks about trou­ bleshooting, and describe the two main approaches that we are currently pursuing.

The two approaches of interest might be termed the "engineer's approach" and the "techni­ cian's approach . " The engineer's approach rests on a detailed tracing of cause and effect in order to find where the causality chain breaks down. The technician's approach eschews this time­ consuming effort (except as a last resort) and in­ stead relies on experience to suggest likely candi­ date faults to be investigated directly.

Let us use the example of the air compressor to contrast these two approaches. Suppose the stated problem is that the compressor can no longer power several air tools that it normally can drive. The engineering approach might begin by tracing the electrical circuits to ensure that the motor is receiving the correct voltage and cur­ rent. Assuming that it is (and that the belt con­ necting the motor and pump is in place), the next step might entail checking the volume and pressure of air output from the pump unit. An inadequate output would pinpoint the pump as a likely suspect, and an investigation of it would continue in the same vein. In contrast to this approach, a skilled maintenance mechanic familiar with the air compressor knows that the reported symptoms are often caused simply by a lack of lubricating oil in the crankcase of the pump . He would fill the crankcase immediately, and if the air compressor was then able to power the usual air tools he would assume that his suspicions were correct and that the problem was now corrected.

It seems clear that a compute~ based consul­ tant needs to be able to employ both of these ap­ proaches, and be able to switch between them when appropriate. An implementation of the engineering approach has begun in a simple way; it relies on a simulation model of the equipment to suggest a se­ quence of tests or observations corresponding to a sequence of causes and effects. An indication of a malfunctioning component is obtained whenever an observed effect differs from the effect predicted by the simulation. An alternative to this imple­ mentation is suggested by Shortliffe'sl 4 rule­ driven system, in which each rule corresponds to a simply stated fact or rule of thumb. We are cur­ rently investigating the extent to which some sort of rule-drive system can be adapted to mechanical troubleshooting.

The domain of electromechanical machinery is an extraordinarily difficult one in which to do automatic scene analysis. Equipment and compo­ nents usually have only a limited range of hue and saturation values. Visual texture similarly is very limited; specular highlights are often complex, and can vary depending on such vagaries as "oil-canning" of sheet metal parts; shadow pat­ terns can be very complex and depend in compli­ cated ways on the particular stage of assembly; and, finally, machines and components assume a very wide variety of shapes. Indeed, scene anal­ ysis in the domain of machinery is likely to be more difficult than, say, in the domain of offices or even landscapes, because in the latter two do­ mains there is a much richer variety of perceptual clues. For these reasons, we have elected to ap­ proach the vision problem by capitalizing on prior knowledge of visual appearances and geometric re­ lations, and by limiting our aspirations at the outset to a set of subproblems that are both im­ portant and tractable.

We have implemented thus far several modular vision packages to perform specific tasks. One interesting module is a tool recognlzer that can accept limited semantic descriptions of tools, build a model of the tool from the description, and subsequently use this model to discriminate an isolated hand tool from a set of alternatives. Let us outline briefly how this is done.

Consider the open-end wrench shown in Figure 5. This wrench is a member of a large class of

hand tools characterized by one or more connected shafts or handles, with an optional "business end" that is applied to a fastening; screwdrivers, nut­ drivers, hammers, and many varieties of wrenches fall in th~s class. The basic operation of the tool recognizer is first to find a shaft, or han­ dle, based on typical aspect ratios for tools, and then to concentrate attention on the end of the shaft in order to determine the tool type. Model information about the endpiece is provided in ad­ vance by an operator, who may use a circumscribed rectangle (as shown in the figure) as a reference for his description. In the case of the open-end wrench, the operator could specify that the end­ piece has a convex curved outline between its up­ per left (UL) and lower left (LL) endpoints, is again curved convex between the upperright and lower right, and is U-shaped between the upper left and upper right endpoints. Relative scale in­ formation would typically also be added to ensure that various parts were reasonably proportioned.

In operation, after finding a tool shaft or handle, the tool recognizer module constructs a loose bounding rectangle in order to determine whether:

1. an endpiece is attached to the end of the shaft/handle;
2. the end of the shaft/handle is connected to another shaft/handle; or
3. nothing is attached to the end of the shaft/handle (e.g., the ends of an allen wrench, the free end of a tool handle).

Coarse size and shape tests are performed to elimi­ nate broad classes of tools. For example, hammer­ like tools are easily distinguished from screwdriv­ ers and wrenches on the basis of aspect ratio. In the case of an endpiece, the loose rectangle also gives gross upper, lower, left, and right limits, which are then refined by shrinking the rectangle to a minimum enclosing size. Figure 6(a) illus­ trates an intermediate processing step using a combination wrench as an example; the program has located the tool shaft, and has circumscribed a tight bounding rectangle around one of its ends. On the basis of this information, the tool may be an open-end wrench, a box-end wrench, or a nut­ driver. Included in the model information for these tools are tests for distinguishing features possessed by the tools. For example, the open-end and box-end wrenches have an opening in the center of the endpiece, whereas the nutdriver is solid in the center. The program generates a square grid of points within the bounding rectangle and deter­ mines whether any of these points is within the brightness range of the background region (Figure 6a). Since some of the points pass the test, the nutdriver is ruled out as a possibility and the program looks for a test which will distinguish the open-end wrench from the box-end wrench. Using the test "Open-at-top," the program attempts to scan an uninterrupted straight line from the center of the bounding rectangle to a point on the top segment ~f the rectangle, as shown in Figure 6b.

Using the distinguishing features tests, the program has refined its hypothesis about the iden­ tity of the tool to a single candidate, the open­ end wrench. It now attempts to verify this hypoth­ esis using model information which describes the shape of the endpiece, in this case by finding the curved convex sides and the U-shaped opening between the upper right and upper left endpoints of those sides.

It is interesting to contrast this approach to tool recognition with a brute force template matching approach. On pragmatic grounds, tem­ plate matching is not very attractive simply be­ cause of the variety of tools (and of sizes of tools), the large number of translations and rota­ tions that a tool can assume, and the inadequacy of template matching if a tool is partially oc­ cluded. On conceptual grounds, the approach out­ lined above is interesting chiefly because of the ease with which new tools can be described by their functional characteristics. In the wrench example, the description is a primitive attempt to say that "anything that can be used as a wrench is in fact a wrench." Tools are a particu­ larly good domain in which to pursue this philoso­ phy because they are artifacts with clear func­ tional purposes.

Two other interesting vision modules entail the ability to point to specific components of machinery. Both rely on. an underlying geometric model of the equipment at hand. The first module enables the consultant to answer requests of the form "Show me the X" by pointing at . X with the laser rangefinder. This is accomplished using a hidden surface algorithm of a very simple variety to locate the outline of a visible surface of the desired component. Once the surface has been de­ termined, a little care is needed to guard against the possibility of pointing at the component by pointing through a hole or a concavity. (For ex­ ample, we would not want to point at a doughnut­ shaped part by pointing at the hole.) A simpli­ fied form of the medial axis transformation is used to find a thick region of the part that can serve as a target.

The second pointing ability is intended to allow the novice technician to ask questions of the form "What part is this?" by pointing at the unknown part himself. We currently use a wand with a small light at the tip to simplify pro­ cessing, yet retain a natural form of pointing for the novice. A ray in space is defined by the wand tip and the camera lens; intersecting this ray with a geometric model of the equipment pro­ vides the information needed to answer the ques­ tion.

The modules described above have some direct extensions that we expect to pursue in the near future. For example, the pointing modules rely heavily on ·the use of geometrical models of equip­ ment. Using these models, we expect to develop means for finding and determining the orientation of a machine or component of interest to the nov­ ice. An open question centers on the extent to which range data will simplify the problem. We have already devoted a good deal of attention to a formalism for combining multisensory data;ls we will need to explore the ease of applying the for­ malism to the complicated collection of shapes typical of most machinery.

Having said something about our plans for vi­ sion, we should perhaps also mention that we are not planning in the near future to use vision to answer fine grained mensuration questions like "Are the pulleys aligned sufficiently well?" We expect that most realistic questions of this form will tax the resolution of our transducers and prefer not to devote our energy to this class of problem.

Language Communication

We were persuaded at a very early stage that natural language communication would have to be an integral part of a mechanic's consultant. Aside from the unfamiliarity a mechanic presumably has with a computer terminal, it seems unreasonable to ask, say, an auto mechanic to crawl out from under a car in order to ask how to replace aU-joint.* Thus, our ultimate goal is to allow the novice to use natural English speech to talk to the computer based consultant. Symmetrically, a second goal is to enable the consultant to talk to the novice using ordinary speech. In recognition of the dif­ ficulty of these ultimate goals--chiefly the first one--we have set as intermediate goals the develop­ ment of more restricted language components that will still allow our experiments to proceed.

Our current language ability rests heavily on a commercially available device known as a phrase recognizer. t The phrase recognizer is able to rec­ ognize an isolate speech fragment up to two sec­ onds long, once the device has been trained by lis­ tening to the novice say each phrase (or word) several times. In our experimental work, a typi­ cal vocabulary of roughly 50 words is evenly di­ vided between object names and control words . Con­ trol words include items like "Why," "How," and "Show-me-the," while object names are mainly part

names.

To enable the computer based consultant to talk to the novice, we use a commercially available phoneme synthesizer. § The synthesizer accepts a sequence of phoneme specifications from the computer and converts these to audible form, producing speech output. A programmer-defined output vocabulary is implemented by selecting a phonemic representation for each word; once this has been done, that word (which is reasonably un­ derstandable) may be used in any context. There is thus no intrinsic limit to vocabulary size, and the system is much more convenient and com­ pact than say, a direct digital representation of acoustic waveforms.

* Our intuition recently received some support when we learned of a simple program providing diagnos­ tic advice to auto mechanics. When field-tested by a major auto manufacturer, it was found to be "A disappointment, because the men wouldn't go near a keyboard."

tOurs, the VIP-100, is manufactured by Threshold Technology, Inc. Other suppliers are also en­

tering the market.

The combination of phrase recognizer and phoneme synthesizer has been notably useful in al­ lowing us to experiment with fragmentary versions of a consultant system. They permit us, at the very least, to gain some intuition about the "live" behavior of such a system. On the other hand, the phrase recognizer in no sense "under­ stands" linguistic content, there is a very lim­ ited ability to handle even simple sentences, and there is no representation of an utterance other than its recognition as one of a limited number of alternative phrases. Consequently, we view the current capability as being only an experimentally useful (and easily achieved) interim one, and are devoting our energy to developing a_ more adequate natural language component for the consultant.

Our immediate goal in this regard is to im­ plement a language component able to deal with natural text input. Simplifying matters for discussion purposes, accomplish this: tion facility, an we need three things in order to a sentence-by-sentence transla­ internal representation of input sentence meanings, and a discourse analysis model. Sentence-by-sentence translation is driven by Pax­ ton's "best-first" parser.l 7 This parser was originally intended for, and indeed is used in, a natural speech understanding system, but it has been modified to acc!ft text input while work in acoustics continues. Accordingly, let us focus attention on the internal representation, which doubles ·as a target language for the parser.

We are planning to use a semantic net repre­ sentation that follows roughly along the lines suggested by Normanls and Simmons.l9 Let us use a few simplified examples to convey both the gen­ eral approach and its application to our particu­ lar needs.

A semantic net representing some information about simplified air compressors is shown in Fig­ ure 7. Each node of this particular net represents an object or a set of objects, and each arc a binary relation between nodes. The upper nodes indicate that the set of air compressors, of tanks, of pumps, and of motors are each subsets (the S relation) of the set of all machines. The set of air compressors is partially defined--we will say delineated to indicate the definition is only partial--by the subnet enclosed in the box. This subnet represents a prototypical member C of the set of air compressors; we use ed to show that C is the delineating element of the parent set. Compressor C is shown as having three parts (HAP means has-as-part): T, P, and M. The e (element-of) arcs emanating from these nodes indi­ cate that they are respectively elements of the set of tanks, pumps, and motors. Thus, the por­ tion of the net we have discussed so far repre­ sents the fact that a typical air compressor is composed of a t.ank, a pump, and a motor, and that all of these objects are machines. The remaining portion of the net represents the fact that there is a particular compressor called COMPl (it is an element of the set of all compressors), and that it has as parts*a particular tank Tl, a pump Pl, and a motor Ml.

§We use a Votrax voice synthesizer, manufactured by Federal Screw Works.

** We plan to extend the system from text to speech

at a later date.

Nodes can also represent abstract entities like relations. The net in Figure 8 is a repre­ sentation of the "drive" relation (drive in the sense of "to power") between two pieces of machin­ ery. It shows that drive relations are a subset of the family of all relations, and it delineates drive by displaying a subnet of a typical drive relation D. The delineation shows that drive re­ lates two objects, a driving object M' and a driven object P'. By following thee arcs out from these nodes, we see that M' must be a motor, and that P' must be a component that can be driven. The net also shows that pumps are among the set of components that can be driven, whereas motors are not.

* We are simplifying some bookkeeping details needed in practice to associate Tl with all tanks, Pl with all pumps, and so forth. Concep­ tually, the reader can think of COMPl and C as having matched subgraphs.

If we are now given the question "What drives the pump?" the net of Figure 8 provides the semantics that enable us to parse the ques­ tion satisfactorily. In particular, it shows that drive is a binary relation, and that "pump" is something that can be driven. (In contrast, the question "What drives the motor?" would be re­ jected because motors are not shown by the net as being drivable.) The "scratch net" at the bottom of Figure 8, which is essentially a copy of the delineation subnet, represents the parsed ques­ tion "What drives the pump?".

Although the net in Figure 8 enables us to parse the question, it does not contain enough in­ formation for us to construct an answer. For this, we need the additional information .repre­ sented in Figure 9. Figure 9 shows fragments of

the previous two nets, but we have augmented the delineation of the typical compressor C. The new delineation uses the relation D' (an element of the set of drive relations) to include the infor­ mation that motor M drives pump P. Using this in­ formation, it is possible to match the scratch net in Figure 8 with the net partially shown in Figure 9 and thus to construct the correct answer. Further details are available in a forthcoming pa­ per by Hendrix.ao The preceding discussion sketches our current design for semantic representation. The third part of the language component has to do with the use of a discourse history to provide the context in which to understand a sentence--for example, to resolve references. In the preceding example, we passed over the issue of how "the pump" is associ­ ated with the particular pump at hand (say pump Pl of Figure 7). We expect to rely on an accumula­ tion (or a summary) of previous scratch nets, with links to procedural nets, for this purpose, but do not yet have a detailed design for this aspect of the system.

A Brief Example

We are still some distance away from having a smoothly running consultant system containing all the functional components described above. Never­ theless, we include the following fragmentary ex­ ample, from a transcript of a live voice experimen­ tal run, in order to give a little of the flavor than a more nearly complete system would have. For ease of comparison, we again have specified the task to be "Assemble the air compressor."

System: Try to achieve assembled aircompres­ sor. Novice: How? System: The top level plan is ... install pump ... install pumpbrace ... in­ stall pumppulley ... install belt .·. install belthousingcover System: Novice: System: Novice: System: System: Novice: System: Novice: System: Novice: System: Novice: Ready for questions. Show me the pump. (Laser beam moves to point at pump.) Ready for questions. What is this? (Pointing with wand at tank platform) Tank platform. Ready for questions. None. Install pump. How? Connect pump to platform. How? Get 4 1&1/4-inch #5/16 hexagonal bolts and 4 1~/16 flat washers and 4 size 5/16 hexagonal nuts. OK.

In this example, the system has composed a hierarchical plan for transforming an initial state of the compressor to the desired final state of complete assembly. The novice then exe­ cutes the plan, while the system keeps track of the current state by using the procedural net rep­ resentation of the plan. The simple replies of "OK" and "How?" (or their equivalents) tell the system either to move to the next step at the cur­ rent level of detail, or to expand a step into a number of more detailed actions. If the novice were to ask "Why?" some step was suggested, the system would use the procedural net to construct an answer that might involve either supergoals of the current step or subsequent steps of the cur­ rent subplan.

Future Problems

It is obvious that a great deal of work re­ mains to be done to bring the individual compo­ nents of the system to a uniformly high level of sophistication, and to integrate these components into a smoothly running system. It is interesting to note, however, that even in its present rudi­ mentary state the system exhibits a surface behav­ ior that impresses outside observers favorably. (We will forgo the opportunity to speculate on the implications this observation has for the pre­ sumed mechanisms of intelligence.) In there­ maining few paragraphs, we will briefly discuss the outlook for continued progress on the consul­ tant system.

It seems to us that the situation is reason­ ably optimistic, at least as far as the individual components of the system are concerned. Planning for assembly/disassembly appears to be quite well in hand, and the only real uncertainties center on how much work is needed to encode how much detail for machines of interesting complexity. Planning for troubleshooting is less advanced, but there is a good stock of ideas available and at least one demonstration program (Shortliffe's MYCIN program) to encourage us that quite sophisticated trouble­ shooting abilities are within reach. Vision, how­ ever, is· clearly difficult in the domain of real machinery, and a general and powerful capability here is not likely to be forthcoming in the near future. However, by relying heavily on geometric and other models, which we assume would typically be available, it appears that a vision component can be evolved that, though limited, would still be very useful. Natural speech understanding is not yet a reality in our system, but good progress is being made on that topic at a number of labora­ tories,l0, 11, 1a, 13 By carefully matching our in­ terim language designs (e.g., the semantic net representation) to this body of work, we expect to be able to make use of forthcoming speech under­ standing systems with minimum effort.

It is less easy to assess the difficulty of integrating all of the functional components into one smoothly running system. One obvious problem is the current multiplicity of models; at the moment, we have at least one model each for assem­ bly planning, troubleshooting planning, vision, and language. Some of these models may be com­ bined--for example, the procedural net may play a role in modeling a discourse history. The funda­ mental problems, however, are much deeper, and in­ volve issues of coordinating very diverse knowl­ edge sources,ll of thinking versus acting, and of human novice effort versus "expert" computer ef­ fort. All this persuades us that computer based consultant systems are likely to continue to be a fruitful domain for artificial intelligence re­ search, in addition to offering promise as a means for deploying knowledge usefully in an increas­ ingly complicated world.

Acknowledgment

The ongoing project described in this paper is of course the work of a large number of people. I would like to especially acknowledge the contri­ butions of G. Agin, A. E. Brain, B. Deutsch, R. 0. Duda, R. E. Fikes, T. D. Garvey, J. Gaschnig, G. G. Hendrix, N. J. Nilsson, W. Paxton, E. Sacerdoti, G. L. Sutherland, J. M. Tenenbaum, and H. C. Wolf.