Impression Monitoring in Evaluation-Oriented Dialog - The Role of the Listener's Assumed Expectations and Values in the Generation of Informative Statements

IMPRESSION MONITORING IN EVALUATION-ORIENTED DIALOG The Role of the Listener's Assumed Expectations and Values in the Generation of Informative Statements

Anthony Jameson

Department of Social Psychology, University of Nijmegen 6500 HE Nijmegen, The Netherlands

ABSTRACT

A prototype dialog system is presented which specializes in responding to the questions of a user who is assumed to be attempting to form an evaluation of a given object. On the basis of explicit assumptions concerning the evaluator's standards and prior expectations, the system goes beyond the direct answering of the questions by selecting additional comments according to their anticipated impact on the evaluator's impressions of the object. The system may be positively or negatively biased in its select1on of comments; taking into account the (possibly different) bias which it assumes the evaluator to ascribe to it, it anticipates how the fact that it has failed to make certain comments is likely to be interpreted. The system's central concepts are also used to quantify the notion of the relatedness of a given comment to a given topic and to guide the selection of connectives and sentential adverbs.

A small boy in Texas had the habit of asking strangers where they were from. "You shooldn't ask them that," said his father. "If a man's from Texas, he'll tell rou so himself. If he's from anywhere else, he 11 be ashamed if you ask."

The research reported here was conducted in part at the Research Unit for Information Science and Artificial Intelligence at the University of Hamburg, which is supported by the German Ministry for Research and Technology. The author is indebted to the members of that group, as well as to Adam Mehrdad, for helpful discussions.

1. What about facilities outside the room THERE'S A KITCHEN

AND THERE'S A BATHTUB

AND BY THE WAY THE ROOM IS VERY LARGE

1. Is there a washing machine and a dryer NO

BUT THERE IS A WASHING MACHINE

1. ~g the room on a quiet street? IN FACT UNFORTUNATELY THERE'S A LOT OF NOISE FROM THE STREET
2. Is there a lot of noise from other rooms WHY NO

This anecdote illustrates three points which should be taken into account in the design of any dialog system which supplies the user with information which may be relevant to the evaluation of a particular person or object:

1. Human speakers frequently volunteer unsolicited comments which have consequences for the evaluation of an object under discussion.
2. The selection of such comments is influenced by the nature of the speaker's bias, e.g. a desire to present the object in a favorable light.
3. Listeners take these facts into account when interpreting such comments (or their absence) - so much so, in fact, that it is wise for the speaker to anticipate such interpretations.

The present paper describes IMP, a dialog system designed specifically as a restricted prototype which permits the exploration of these topics without the overhead associated with an NL system of broad capabilities. The system en~ages in what may be called evaluation-oriented d1alog: It takes the role of an informant responding to the questions of an evaluator whose sole dialog goal is assumed to be the assessment of some object, e.g. with a view to making a decision concern1ng 1t (cf. the example dialog in Figure 1).

Dialogs of this general sort occur in many everyday situations - e.g. personnel selection interviews and discussions with salespeo~le. The evaluation-oriented dialogs which IMP can handle are quite restricted in terms of the number of speech act types and dialog goals involved, as can be seen from the description of its top-level control procedure in Figure 2.* This simplificat1on makes it unnecessary for the system to reason explicitly about dialog goals and ways of realizing them (cf., e.g. Cohen, 1978), although the system does consider carefully what impact its utterances will have upon the listener.

Open-ended question; no direct answer possible.

Availability of oven not important enough to warrant more compiex comment.

Prefaced with and because resulting impression shift is in same direction as previous one. Presence of washing machine not worth mentioning; it corresponds more or less to expectations.

• Though not directly related to question topic, worth mentioning because of large anticipated impression change. (If the system were not positively biased, it would have likewise spontaneously mentioned that the room is noisy.)

More specific question, direct answer possible.

Anticipated to produce slight downward impression shift. This comment, now worth maKing because of narrower question scope, should shift impression back upward,

Direct answer does not betray how noisy room is. System would prefer to make no additional comment but sees silence would be inconsistent with its projected objectivity. Question concerning closely related topic. that

Expression of surprise at question (cf. Lakoff, 1973): If this were the case, an objective informant would already have mentioned the fact when responding to the previous question. (To be sure, the system itself would not have done so, because of its positive bias.)

Fig~ 1. Unedited dialog in which IMP (output in capital letters) responds to questions concerning a particular room offered for rent. (The system's bias is positive here, but it attempts to maintain a neutral image.)

X 5

8

THE REPRESENTATION OF VALUES AND EXPECTATIONS

IMP is supplied in advance of each dialog with an evaluation form (Figures 3 and 4) 1 which can be viewed as a particularly explicit var1ant of a type of scale which one frequently sees used in practical contexts which require systematic evaluations (e.g. the refereeing of conference papers). The system behaves as if 1t believed that the evaluator possessed this same form and was asking questions in order to fill it in so as to be able to estimate a numerical rating for the object along the dimensions corresponding to the var1ous scales. It is assumed that the evaluator will not in general ask questions about all of the items in the form, but will use the frequency information in it to make estimates concerning the scales not covered.

The basic idea underlying this metaphorical conception of the evaluator's judgmental processes is that the subjective evaluation of an object is an additive function of the perceived value and likelihood of its possession of various attributes.** This idea fits into a long tradition in normative and descriptive research on attitude and judgment (see, e.g., Fishbein & Ajzen, 1975). It could be made more realistic 1f results concerning, e.g., the way people deal with small probabilities were taken into account; but it is doubtful that such changes would noticeably improve the app~o~riateness of IMP's re~pon~es or increase its val1d1ty as a model of a na1ve 1nformant.

* Winograd's (1983) language DL is used here (somewhat elliptically) to describe the essential workings of the system. Procedures whose internal structure is not relevant to the issues of interest here are not described; most of these are realized in IMP using ad hoc methods which make heavy use of specific hand-coded database entries.

Invocations of such procedures are underlined in the figures and marked with an x in the right margin; all other underlined concepts are introduced in the same figure or in the figure whose number a~pears to the r1ght.

As actually 1mplemented, the ~rogram comprises 50 LISP/FUZZY procedures and requ1res several seconds to respond to a question.

8 6

** The values specified are conceived as lying on a single interval scale. It is presup~osed that the items are independent of each other 1n the sense that neither the values nor the frequencies associated with the possibilities for a given item depend on what possibilities are realized for other items.

The evaluation form of Figure 3 is an abbreviated version of one written largely intuitively by the author. The general values and expectations it expresses are ascribed by IMP to any evaluator who questions it on this topic and are not revised on the basis of the evaluator's behavior during the dialog. For use in future research, a wide variety of more empirical techniques are conceivable for obtaining scales for particular user groups or even for individual users, building, e.g. on the work of Rich (1979) or on techniques developed by decision analysts (surveyed by Slovic, F1schhoff, & Lichtenstein, 1977).

SELECTING COMMENTS - OVERALL STRATEGY

Figure 5 shows how in IMP the task of generating appropriate unsolicited comments is conceived as a search through the evaluation form for items which warrant some comment other than the pseudo-comment '*silence*'· This search could undoubtedly be made more efficient and plausible if additional heuristics were introduced to make it mor.e selective, e.g. by making early reference to the items' overall importance, which is at present used only for the deta1led rating of individual comments (cf. Conklin & McDonald, 1982).

4

X

6

8

X

4

4

~ 4 X

2

X

Figure 6 describes the processing performed for each 1tem in the form that the system inspects. IMP has no interesting methods for constructing new comments on a g1ven item; of interest is only the way in which it selects the most appropriate one from a set of candidates which is assumed to be representative of the infinite set of comments that

How .would (THERE'S A LOT OF NOISE FROM THE STREET) be interpreted? ...

So it would be inconsistent with my projected bias.

Literally.

OK.

Reducing the remaining possibilit1es from (B C) to (C) for Item 2.1.

>>> IN FACT UNFORTUNATELY THERE'S A LOT OF NOISE FROM THE STREET

Eig~ 1. Excerpt from IMP's trace for its third response in Figure 1, showing why it made an additional comment even though silence appeared at first to be the most attractive alternative. (Trace edited to incr.ease legibility.)

could conceivably be made. Its overali strategy is first to select the comment which, if interpreted literally, would have the most desirable effect on the evaluator's impressions, and then to take into account the way this comment would in fact be interpreted. The system anticipates this pragmatic interpretation both in order to keep track of the evaluator's changing beliefs concerning the object and to help filter out comments which would betray its bias (as shown in Figure 7).

RATING POSSIBLE COMMENTS

Impression Changes

The kernel of IMP is the criteria which it uses to assign a numerical rating to a possible comment, assuming that the comment will be interpreted literally. The central factor which it takes into account is the comment's anticipated impact on the evaluator's corresponding impression of the object (Figures 8 and 9). The motivation underlying the concepts of expected value and uncertainty introduced here can best be seen if one considers how changes in these quantities are related to the dialog goals of an informant.

0~ the one hand, any informant is likely to be interested in shiftin~ the favorability of the evaluator's impression 1n some direction either ~enerally upward or generally downward (if the 1nformant is biased) or in the direction of the truth (if the &oal is to present an accurate profile of the obJect). In any case, the change in expected value should enter into the formula for rating a comment (Figure 10).

On the other hand, even a comment which merely confirms that an expected value is in fact accurate may be considered worth making if considerable uncertainty previously existed with respect to the item. A reduction in uncertainty puts the evaluator in a better position to make decisions, anticipates questions that might otherwise have to be asked and answered separately, and is ~enerally felt to be a part of polite, ~ooperative d1alog behavior.

4X

The relative weight assigned to these two factors by a given informant can in principle vary greatly as a function of various features of the dialog situation. At present, they are given equal weight, so that they both have noticeable effects on the system's behavior. Their sum is regarded as the 'benefit' which would result from making the comment in question.

Effort

If the system is not to break into tedious monolog after each question, it must have some notion of the effort involved in making and understanding a comment. The simplest solution would be for it to subtract from the rating of every comment except '*silence*' some number such as 50 (the exact one chosen depending on the degree of volubility desired). But this would not do justice to the fact that some utterances are more detailed, complex, roundabout, or unnatural than others (cf. Grice's [1975] Maxims of Quantity and Manner) and must thus be considered to be associated with more effort. IMP makes no contribution to the difficult problem of defining and quantifying such a notion (see, e.g. 1 McCawley, 1978); it simply uses stored rat1ngs which assoc1ate different degrees of effort with the various possible comments.

Question-Relatedness

Even when volunteering comments which have not been specifically requested, the system should give preference to comments which are directly or indirectly related to the topic of the evaluator's latest question (cf. Grice's Maxim of Relation). There are a variety of reasons why speakers tend to do this. For exam~le, wnen a question has been asked about a g1ven topic which, according to the evaluation form, is relatively unimportant, this fact suggests that the present evaluator may attach more importance to this topic than was or1ginally expected. Although this need not be the case, it may be worthwhile to devote increased attention to such topics in order to take the possibility into account .

IMP's criteria for choosing comments are designed in such a way that the system's ratings will satisfy several constraints which appear generally reasonable in view of considerations such as these.

Constraint 1. The less imRortant the t.QRic of E.. question is, the more strongly the benet'its assoclated With l! OirectlY.. related- -comment should be 1 maen1ffedT berore-be1ng wei~ against the eTfortlr.voiVed in ~g the coiiiiiient. Tfierelevant notion of the general 1mportance of a topic can be captured quite naturally as in Figure 4 above in terms of the uncertainty of the evaluator's initial impression concerning it, since this reflects the likely impact that knowledge concerning the topic will have on the evaluation of an arbitrary object.

. Constraint .Z,. This guestion-dependent magnif1cation should not be so stron' as to cancel out all dlfferer;ces-IntJie 1ntr1ii'STC lJ!lROrtance 2f topics. If the magnifiCation factor used were Simply the reciprocal of the topic's relative ~mportance, the result would be be that about as many comments would be volunteered in res~onse to unimportant questions as in response to 1mportant ones. So that the system will instead take both factors into account, the square root of this ratio is used instead (more ~enerally, the kth power, where k lies between 0 and 1 and reflects the system's responsiveness to the question topic.

Constraint 3. The less closelTh;~eg§mment ~ related tQ tne guestTQr; j&p_ll,,h lli.Qngly 1ts beneffts ShOuld be mag~. he degree of quest1on-relatedness--of~he toplc of a comment can be captured quantitatively with reference to the most specific topic that the two both have some bearing on, as in Figure 5. It can be seen as reflecting the extent to which the question asked would have to be broadened before it would include the comment within its scope. Multiplying the magnification factor for the comment by its question-relatedness has the consequence that there will be no magnification at all for comments which have no specific relationship to the question.

Constraint !!,. The ~ associated 't!.i.t.h .a comment should be considered greater ~ ~ closely_ il U! related t& ~ guestion ·· Although Constraint 3 takes into account the fact that unrelated topics have not been singled out for attention by the questioner, it does not do justice to the fact that such a comment shifts the evaluator's attention away from the l~test topic introduced. This shift is conceived 1n IMP as affecting the associated effort multiplicatively, with the reasonable consequence that the system is even less likely to choose a comment requiring a large amount of effort for a topic which is not· directly related to the question.

CONNECTIVES AND SENTENTIAL ADVERBS

Whenever a sequence of evaluation-relevant statements is made, it is usual to preface each one (except perhaps the first), with words which announce to the listener in advance what sort of impression change is about to occur. The rules that IMP uses for the selection of expressions like but, by the way, and unfortunately make reference to a comparison of the shift in expected value produced by the present comment with that produced by the preceding statement, taking into account the question-relatedness of their respective topics. Some conceptual apparatus such as the one introduced above for independent reasons is required if the use of such expressions in evaluation-oriented statements is to be explicated with a satisfactory degree of precision (cf. Weydt, 1979).

CONCLUDING REMARK

When IMP's actual and projected biases diverge, as in the examples given here, its behavior smacks of deception, and it mar be asked what practical consequences the deve o~ment of systems of this sort might have. But as d1alog systems continue to be accorded more initiative in their interactions with users, questions concerning their dialog motivation will have be investigated explicitly and publicly, whether the aim is to formulate realistic norms or simply to prevent discrepancies from arising between the systems' actual motivation and their users' image of it.