Comparing the Conceptual Systems of Experts

1 Sources of Dissent in Conceptual Systems

Computer elicitation of entity-attribute, or repertory, grids has proved to be a powerful technique for acquiring the declarative knowledge structures of an expert in a domain [Shaw and Gaines, 1983, 1987, Boose 1984, Boose and Bradshaw, 1987, Diederich et al, 1987, Gaines, 1987a]. Entity-attribute grids are used to elicit from the expert two types of information about a particular sub-domain: the distinctions made between relevant entities in that sub­ domain; and the terms used for these distinctions.

In a well-established scientific domain it is reasonable to suppose that there will be consensus among experts as to relevant distinctions and terms-that is, objective knowledge independent of individuals [Popper, 1968]. However, the "expert systems" approach to system development has been developed for domains where such objective knowledge is not yet available, and the primary sources of knowledge are the conceptual structures of individual experts [Gaines, 1987b]. When multiple experts are available for a domain where a consensus has not yet been reached, it is important to be able to compare their conceptual structures, both among themselves and with those of potential clients for the resultant knowledge-based system.

Studies have been made of extending performance systems based on entity-attribute grids to allow multiple knowledge structures from different experts to be combined without distortion, enabling "dissenting opinions" to be requested [Boose, 1986]. In some cases, however, apparent dissent between experts is due to conflict in using the same terminology for different concepts. In addition apparent difference in viewpoints can result from the use of different terminology for the same concept. Figures 1 through 4

In Figure 3: correspondence arises if the conceptual systems assign different terms to the same distinction.

In Figure 4: contrast arises if the conceptual systems assign different terms to different distinctions.

Consensus shows a shared conceptual sub-system. Conflict shows possible dissent over the application of a concept in a domain since the same term is being used in different ways. Correspondence shows possible dissent over the appropriate term to be used for a concept Contrast shows conceptual sub-systems that are not shared, and cannot lead to direct dissent. It may indicate different areas of expertise.

The following section describes a methodology for eliciting and processing entity-attribute grids that derives this classification for groups of experts and clients.

2 Comparing Conceptual Systems

Any comparison of conceptual systems necessarily involves approximation since a complete conceptual system may involve indefinitely complex relations and different concepts will never be identical in all respects. However, in the initial phases of knowledge acquisition, highlighting gross similarities and differences is itself valuable in promoting directed discussion among experts and clients that can lead to the exposure of more subtle relationships. As a start one wishes to elicit the major distinctions that an individual uses in a domain, the terminology used for them, and the relation of such distinctions and terminology to those of others.

Entity-attribute grid elicitation is an effective method for eliciting major distinctions and terminology in a domain [Shaw, 1980]. It is an extensional approach in which individuals are asked to specify a set of entities in a domain, then make distinctions among them, naming the distinctions and classifying all the specified entities in terms of them. The extension of a distinction determined in this way is only an approximation to the underlying concept since critical entities may be missing in the classification. However, both manual and computer-based elicitation techniques attempt to prompt the individual for additional entities to discriminate between extensionally related distinctions (that is making the same, or similar, classifications).

Group comparisons, as discussed in this paper, have similar dynamics-an extensionally apparent consensus or correspondence may be accepted or rejected, and the rejection may be based on the specification of additional entities as counter-examples. Knowledge acquisition is essentially a negotiation process leading to approximations to conceptual structures that are adequate for some practical purpose such as system development

Measures for comparing extensionally defined distinctions may be formulated in terms of a fuzzy sets model of grid data [Gaines and Shaw, 1980, 1986]. The bipolar attributes in a grid are treated as a pair of predicates defining fuzzy sets and the rating of an entity on an attribute is regarded as defining a degree of membership to each of these sets. Following the conventions of Gaines [1976] we shall use ordinary characters for fuzzy sets, boldface characters for fuzzy variables, and denote the degree of membership of a variable to a set by concatenating it to the right of the set. Thus, x is the fuzzy set defined by predicate x (a LHP or RHP of the grid) and the degree of membership of entity e to fuzzy set xis xe. A measure of equivalence between two attributes can then be defined as:

where the and and equivalence operators on the right hand side can be any of the many proposed for fuzzy logic-in the examples in this paper we use Lukasiewicz's operator for equivalence and Zadeh's sigma count for conjunction [Smets et al, 1988]. Other distance measures that are zero when attributes are applied identically to entities and increase continuously with increasing differences in application could also be used.

To apply (1) as a measure of similarity between distinctions it is necessary that they have the same extension-that is, that the group agrees on a common set of entities among which they will each make distinctions. Phase 1 of the methodology consists of establishing this common set of entities and ensuring, as far as possible, that each individual in the group is defining the entities in the same way. This appears straightforward if the entities are well-defined, named concrete objects in the world. However, the concept of an entity turns out to have its own difficulties in practice-if the entities are people, is it important in what mood and what role they are?-if the entities are tools, is it important who is using them and to what purpose? Again, entity definitions are part of the ongoing negotiation process of knowledge acquisition, and tacit disagreement over them can be a source of dissent. The methodology described compares entities also using a similar measure to (1) over the attributes being used.

For the sake of simplicity, we will phrase the following in terms of comparisons between pairs of individuals. The methodology extends simply to a group as discussed later. The measurement of consensus and conflict requires a common terminology for the distinctions and is based on the exchange of grids. One individual is given the grid elicited from the other with attributes and entities specified, but not the values, and asked to rate the entities on the attributes. The measure defined in (1) can then be used to determine how well the distinctions made by the second individual match with those made by the first. Distinctions which match above an upper threshold are in consensus and those that match below a lower threshold are in conflict-those matching between the thresholds are ambiguously related.

Correspondence and contrast between two conceptual systems may be measured by eliciting independent grids from the two individuals and using the measure defined in

(1) to determine for each distinction in one grid the best matching distinction in the other. Distinctions which match above an upper threshold are in correspondence and those that match below a lower threshold are in contrast-those matching between the thresholds are ambiguously related.

As important as the mathematics and methodology is the presentation of the results to the individuals in such a way that they readily visualize the areas of potential dissent and their sources. Figure 5 is a graphic presentation of a consensus and conflict analysis from a grid exchanged between two experts in geographic mapping techniques [Shaw and Woodward, 1988]. The differences are shown in the grid on the left in numeric form and as shading with darker gray indicating greater difference. The sets of attributes and entities have both been sorted so that those with highest matches indicating greatest consensus are at the top of their respective lists. The graphs on the right show the declining match values and may be used to decide where to place thresholds distinguishing consensus and conflict

The presentation of Figure 5 contains sufficient raw and processed data that it may be used for a range of discussions, from identification of apparent disagreements, such as that on the use of the term "linear interpolation" and the entity "probability matching," to a detailed analysis of how these disagreements have arisen. Measures of correspondence and contrast are presented in a similar graphical form with attributes paired from the two grids.

Figure 6 shows the basis for the calculation of the results given in Figure 5. The ratings given to the entities on the attribute "Discontinuous-Continuous" by each expert are shown on the top two lines (1 indicates "Discontinuous" and 5 indicates "Continuous"). The differences are shown on the third line and correspond to those for this attribute in Figure 5 (noting that the entities are re-ordered in Figure 5 and that zeroes are not shown).

To provide a synopsis for discussion by the experts, the results of the comparisons are summarized and presented in the four quadrant diagram exemplified in Figure 7 (the 70!80

levels are chosen because they symmetrically separate the no match level of 50 from the total match level of 100-a match level of 0 corresponds to a perfect inverse match).

The consensual items in Figure 7 are simple to deal with­ an obvious question is "to what extent can a knowledge­ based system be built based only on these well-agreed distinctions?" The conflicting items can be looked at with the reverse implication, "to what extent can a knowledge­ based system be built not using these ill-defined distinctions?"

If some of the distinctions are agreed to be necessary for system development then the basis of the conflict can be examined in greater detail starting with the presentation of Figure 5. Many resolutions of the conflict are possible, but the computer-based methodology can go no further than making the basis of conflict and the possible need for resolution apparent

The corresponding items may again be looked at in terms of their relevance to the problem. In addition their further interpretation often leads to significant insights into the knowledge structures involved. For example, the terms low-level-data­ high-level data and nominal data­ interval or ratio data corresponding may be interpreted as arising from different levels of abstraction.

Questions then arise about whether the levels should be represented in the knowledge structures, "which will be best understood by a client?", and "should one term be used as an explanation of the other in response to a help request?"

Interpretation of the correspondences in Figure 7 is an important knowledge acquisition methodology in its own right. The simple example shown brings to light abstraction hierarchies, physical and social causal links, and other relations between attributes. The methodology has the advantages of group brainstorming in stimulating knowledge elicitation, and captures the results in a highly structured and usable form.

Note that the derivation of consensual, corresponding, conflicting and contrasting attributes is completely algorithmic, based solely on the data in the grids. This derivation is done by a computer program, not a knowledge engineer, and its basis can be demonstrated clearly to the experts and clients through computer output such as Figure 5. Thus, there are no opinions being expressed about the correctness of the use of the attributes and terminology, that the differences highlighted are 'right' or 'wrong.' It is open to the experts and clients to consider, discuss and explain these differences, changing or retaining them as they wish. Conflicts can be retained in the final system if desired by tagging classes, objects and rules with the sources from which they derive.

The methodology has been described in relation to comparisons between two people. However, it scales up linearly with the size of the group, so that n people will be involved in n elicitations, one base elicitation and n-1 exchanges. The following section reiterates the methodology in detail as a practical technique for knowledge acquisition with groups of experts and clients.

3 A Methodology for Comparing Conceptual Structures

Figures 8 through 1 0 show the phases of a methodology for comparing conceptual systems.

• In phase 1, the problem is discussed with the group of experts and representative clients, and a domain is identified.
• The use of computer-based knowledge acquisition is introduced, a specific task is agreed upon, and a purpose for the elicitation developed.
• The first step in acquisition consists of the elicitation of a grid from each individual. This gives each person experience in using the computer-based system and provides an initial set of entities and attributes particular to the individual.
• These grids are processed using graphical clustering methods, and the results shown and explained to the

individual from whom the data was elicited. As a result an individual may extend or elaborate their conceptual system expressed in the grid.

• The group meets together to discuss each set of entities, review and revise the purpose or context of the elicitations, and agree on a common purpose and set of entities which all understand to be used in the next two phases.

Flg.9 Methodology Phase 2

• This common entity set provides the basis for the second phase of knowledge acquisition which consists of the elicitation of a grid from each individual using the set of common entities.
• These grids are again graphically clustered, and the results reported and explained to each expert respectively. Again this may lead to extension and elaboration.
• This data is then processed to identify corresponding and contrasting attributes.

• The third phase of knowledge acquisition consists of the individuals exchanging their grids elicited in phase 2 with eac_h other individual in the group to produce grids in whtch one person classifies the common set of entities using another person's attributes.
• This data is then processed to identify consensual and conflicting attributes.
• Their conceptual systems and the relations between them are then discussed by the group. Consensus is identified. Conflict is resolved, qualified or noted as irresolvable. Correspondence is interpreted as due to differences in terminology or correlations through some identified phenomenon. Contrasts are analyzed as irrelevant or a relevant part of some specialist viewpoint.
• This analysis is propagated through all further phases of knowledge acquisition such as class, property and object definitions, and rule specification. For example, possible conflicts are avoided if possible, and identified and explained otherwise-correspondences are used to generate alternative forms of questions and explanations.

4 Conclusions

The knowledge to be acquired for the development of knowledge based systems is often distributed across a group of experts rather than available for elicitation from a single expert. Group elicitation presents major problems because experts can disagree on the use of concepts and vocabulary, and this disagreement may be tacit causing confusion. This paper describes a computer-supported methodology for knowledge acquisition from groups in which the conceptual frameworks of different experts are compared in a way that makes such disagreements overt and readily identifiable.

The methodology provides facilities for revealing the similarities and differences in the conceptual systems of groups of experts and clients. It can be used to focus discussion between experts, and between them and representative clients, on those differences between them which require resolution, enabling them to classify them in t~rms of differing terminologies, levels of abstraction, disagreements, and so on.

The methodology promotes the full exploration of the conceptual framework of a domain of expertise by encouraging experts and clients to operate in a brainstorming mode as a group, using differing viewpoints to develop a rich framework. It avoids social pressures forcing an invalid consensus by providing objective analysis of separately elicited conceptual systems.

Acknowledgements

Financial assistance for this work has been made available by the Natural Sciences and Engineering Research Council of Canada. We are grateful to many colleagues at the knowledge acquisition workshops for discussions which have highlighted many of the issues raised in this paper. We are particularly grateful to Brian Woodward, John Boose and Jeff Bradshaw for many fruitful discussions of knowledge acquisition systems, and to the anonymous referees for helpful comments.