ANTLIMA - A Listener Model with Mental Images

1 Introduction

The project VITRA which started in 1985 as part of the German special collaboration programme SFB 314, AI f3 Knowledge-Based Systems, examines the relations between speaking and seeing: a completely operational form of reference semantics for what is visually perceived is to be developed. CITYTOUR and SOCCER are two systems constructed in VITRA which- broadly speaking - demonstrate the transforma.tions of visual perceptions into language. Here, we will concentrate on SOCCER and its listener model ANTLIMA (for CITYTOUR cf. [Schirra et al., 1987]).

2 SOCCER and ANTLIMA

SOCCER simultaneously analyses and describes short soccer scenes in German similar to a live radio report, i.e., simultaneously and in an objective manner, to an audience which is not able to see the game. As input, SOCCER receives data which is generated by the motion analysis system ACTIONS from the signal of a video camera (cf. [Herzog et al., 1989]). This Mobile Object Data- MOD- consists of the set of the two-dimensional spatial locations and the velocity vectors of every mobile object perceived in the soccer field from a bird's eye view. At every time quantum, ACTIONS delivers such a set. At present, all mobile objects are perceived as mere ideal points. The MOD implicitly refers to the geometry of the football field and its parts given to the system as Static Background - StaB.

·This is an abridged version; the original text is available as VITRA-Report No. 94;

SOCCER does not know the whole scene at once. Like a radio reporter, it has to consider the events during its occurrence. Therefore, all processing steps have to be done incrementally in a kind of pipelining: a selection of already recognized events is verbalized simultaneously with further event recognition. Beside ACTIONS, the architecture of the system consists, broadly, of the Core System including three components ( cf. Fig. 1, [Herzog et al., 1989], and [Andre et al., 1989]) : the compo­ nent for Event Recognition produces a set of proposi­ tions interpreting the given 'percepts' as instances of spatial and spatia-temporal relations - the former es­ sentially correspond to static spatial predicates like 'be­ ing to the left of', the latter to predicates of motion events, e.g., 'doing a double pass'; the component Se­ lection chooses some of those event propositions while planning the continuations of the report ; the component

Taxi Generation transforms the selected event proposition to an utterance in German coherent to the preceding text, thereby occasionally asking for further information for filling optional deep cases. Whereas the core system recognizes purely spatia-temporal events, SOCCER has been extended by the component REPLAI which recog­ nizes plans and intentions of the observed agents ( cf. [Retz-Schmidt, 1991]).

Figure 1: Extended Architecture of SOCCER

In this presentation, we are interested essentially in another component of SOCCER, namely its listener model ANTLIMA. In order to follow Grice's Coopera­ tive Principle- "Make your conversational contribution such as is required, at the stage at which it occurs, by the accepted purpose or direction of the talk exchange in which you are engaged" (cf. [Grice, 1974]) -, a speaker has to know how his utterance is understood by the lis­ teners in the present context . That means he needs a model of the audience, e.g., to make sure that the listen­ ers are still able to recognize all the relevant information even from elliptic descriptions. Such a model allows him to balance between the divergent demands of economy and completeness - "make your contribution as informa­ tive as is required, but also not more informative than is required"- since the speaker can rate how much explicit information actually is required in the given case, and what may be additionally inferred as implicature.

Correspondingly, SOCCER also needs a component that can construct and maintain a model of the lis­ teners' knowledge of the events that have already been described. This listener model enables the system to continue its description in a cooperative way by antici­ pating the listeners' understanding of the utterance just planned. With these anticipations, the understandabil­ ity and plausibility of that utterance in the context al­ ready known can be rated and used in an anticipation feedback loop to the component Selection for improving the coherence ( cf. Fig. 1) . We assume that the under­ standing of the audience is explained similarly to the text generation of the speaker - by reference semantics: the generation of the report depends essentially on the speaker's perception of the referents. Therefore, the lis­ teners also have to represent mentally the referents of the report, albeit referents which they in fact cannot perceive. As a German linguist wrote in 1969, "the ra­ dio reporter has solved his task only if he describes the reality of a sports event so vividly and obviously to the listener that the listener believes he sees that reality." ( cf. [Dankert, 1969, p. 94]) . The reporter should induce - so to speak - a cinema in the heads of his audience. I.e., if the listeners want to have a 'deep' understand­ ing of an utterance, they should be able - following the approach of reference semantics- to create (visual) men­ tal images corresponding to the speaker's percepts ( cf. [Schirra, 1990a]).

Correspondingly, the listener model of SOCCER, named ANTLIMA - ANTicipation of the Listeners' IM­ Agery-, must be able to construct visual pseudo-percepts - albeit in the limited sense of SOCCER as MOD and StaB. The abstractly described situation must be con­ cretized again. It is our thesis that those pseudo-percepts generated by ANTLIMA correspond to the listeners' visua! mental images, and that they can be used to explain the coherence of the text produced. In the context of lis­ tener modeling, the concept 'mental image' in fact par­ ticipates in two alternative but related frameworks ( cf. [Schirra, 1992]): conceived as the mental representation of the understanding of the previous text, it allows for explaining the success or failure of acts of reference: an NP - be it anaphoric or elliptical - can only be used if it uniquely identifies its referent in the image. For example, the NP 'the goal' in a sentence like 'Miller runs toward the goal' is ambiguous if the context does not provide relevant information about the concrete position and di­ rection of movement of Miller which selects one of the two goals of the soccer field . Therefore, the modeling of the listeners' mental images plays an important role for generating adequate noun phrases, especially in the form of elliptical definite descriptions and anaphora ([Schirra, 1991]).

Conceived as the speaker's anticipation of the listen­ ers' understanding of a sentence still to be uttered, the concept 'mental image' makes possible to explain the se­ lection of the predication: whereas the identification of referents makes use of information assumed to be known already by the audience, the selection of the predication to be communicated is concerned with what the audience presumably does not know yet. The corresponding situa­ tion has first to be established in a mental image, thereby elaborating implications specific to the given context. It is this framework and its central question - how to construct a mental image corresponding to some given predications - which we shall examine in this paper .

ANTLIMA starts with the event proposition chosen by the selection component of SOCCER as the core of the planned utterance: in order to be made concrete, this proposition has to be analysed down to the primi­ tive propositions: the definition of the considered event concept, i.e., its subevents with their temporal relations, must be expanded and adapted to the situational con­ straints in order to achieve spatia-temporal · coherence between the required spatial restrictions of the involved objects during each phase of the event and the context ([Schirra, 1990b]). Further constraints given by optional deep case fillers also have to be associated to the appro­ priate phase of the event ( cf. [Marburger and Wahlster , 1983] and [Sablayrolles, 1991]). The construction of a corresponding dynamic mental image is performed in two steps: first, the duration of the elementary subevents are fixed; this results in the propositional elementary struc­ ture, the temporally ordered sequence of sets of elemen­ tary spatia-temporal relations. Each set restricts the scene for exactly one instant (time quantum), describ­ ing, so to speak, a 'snapshot'. This data is transformed in a second step to a sequence of MOD relative to the given StaB by fixing particular locations and velocities for all objects considered for each time quantum succes­ sively.

In order to rate the understandability of the consid­ ered continuation of the report, the generated 'mental image' should not be compared coordinate by coor­ dinate with the original percept of SOCCER: we can­ not expect that. the listeners will reconstruct the very sam e pict ure from that selected set. of propositions ac­ tually communi cated , since we usu a lly lose informa tion by transforming an image to a desc ription ( i. e., propo­ sit ions) - information which has to be recovered while recons t ruct ing the image. Out which deviations are to be tolerated and which are not? If, for exampl e, a player is standing in the middle of the right fi eld -- no­ body near him - 50 pixels ' difference norma lly will not matter. T hough, if he stands at the edge of the fi eld , or near some other player , or very close to the ball , even 10 pixels' deviation of his locat ion might change the whole interpretat ion of the scene. Since t he propo­ sitions resulting from the processing of the component. Event Recognition ignore hy definiti on deta ils of t he percept irrelevant for soccer games, A NTLIIVI A firstly has to re-ana lyse its mental images , i .c. , describe them proposit ionally again. The result is a set. of propositions t hat - from the speaker's point of view - represPnts ex­ pli citly the lis teners ' comprehension of the meaning of the utterance properly intended , including a ll implica­ tu res, etc. Comparisons of these propositions with those originally recognized in the percept by the core sys tem, t hose chosen to be communicated, and those proposition­ a lly inferred by ANTLIMA's conceptual analysis lead to severa l sets of differing propositions: correct or wrong understanding, valid or invalid impli cat ure;; , and fitting or erroneous expectations of continuations of the scene a ll can be dist inguished by means of these sets . Sine<' t his theme does not affect the discussion in this report. , we do not deal further with it here (cf. [Schirra, 1991]) .

Finally, plausibility ratings , correctly predi cted con­ t.inuations of the report, and - if necessary -- detec!('(l misunderstandings are g1ven hack to the core systen1 (component. Selection) where ·· while dosing the anti ci­ pation feedback loop- t hey may be used to increase the pragmati c quality of the report. Additionally, the cre­ ation of the menta l image and its re-ana lysis makes it possib le to const ruct. a l·is11al focus of the audience: the re-analysis brings into focus objects not expli citly men­ tioned. In consequence, t he Generat ion component can refer to t hose objects by means of underspecified noun phrases like ellipti cal J cfinite descriptions without loss of cl arity.

3 The Construction of Mental Images

The centra l assun1pt.ion for the const ruction of mental images in ANTLIMA is the foll owing : the audience ex­ pects that the speaker a lways in tends to describe the most. typical instance of an event. For example, if t he speaker is ta lking about player A transferring t he ball to player B the listener will imagine a direct - i.e., linear - transfer of the ball from A to B without any devia­ tions. If there are any differences to the typical case the speaker has t.o express them, or he violates the listener's communicat.ive expectations. Typicality, on the other hand , depends on the current. context.. In our example: if the speaker and the audience mutua lly know that a ha rd wind has an influence on tlw trajectory of the ball the speaker can <tssume that his audi ence will imagine the ball flying in a curve from A to B as the most typ­ ical instance in this parti cular situation. He therefore does not need to mention explicitly this deviat ion of the linear traj ectory.

For constructing an adequate mental image, i.e., a maximally typical representation of the situation de­ scribed - or restricted - by a proposition, we first have to generate the valid distribution of typicality for all possible mental images. In ANTLIMA, such distrib­ utions are called Typicality Potential Fields (TyPoFs) , functions mapping locations to typicality values . 1 They are used to find the optimal positions: all objects to be located (OLs) are moved to a position with maximal typ­ icality by means of a hill-climbing algorithm ( cf. [Yamada et a/., 1988]). Figure 2 illustrates an exemplary TyPoF corresponding to a proposition of the type 'player in front of the goal area'. The TyPoF is represented graphically by means of grey values: the darker the position , the more typical it is. 2 Note that. different starting positions result in different end positions: this influence of the context of course is a very desirable effect which here is given without additional effort. Furthermore (although not demonstrated in the figure) , if the reference object. A of an OL B is also located with respect to a third object, the TyPoF of B is moved with A: B is pulled along with A until all objects have reached their optimal positions .

Each TyPoF encodes the typicality distribution corre­ sponding to one kind of proposition, i.e., it. depends not on ly upon the spatial relation used as predicate but also on the arguments of the proposition . Although the typi­ cal ity distributions of propositions wit.h t.he same spat.ial relation, like 'being in front. of', but. different sets of argu­ ments are more or less simi lar, t.he kinds of objects and especially their dimensionality, size, and shape modify the TyPoF of a particular proposition , stretching it, and

1 In ANTLIMA, the t.ypicalit.y values are from the ratioual interval (0.0 .. 1.0]; the 'T-value' 1.0 is associated with all locations with maximal typicality;

2 In Fig. 2, the black parts in fac t. represent typicality valnes 2: 0.9; t.he propositions used iu the exemplary approx­ imation are shown in the Status-Pane on the right side of t.he figure ; in the window on the lower side of the picture, the final t.wo approximation steps for player S2 (black-2) are traced; for a complete list. of Ty PoFs d. [nlochcr et al., 1992];

I

!

i

I

I

J

adapt.ing it. t.o the shape of t.he objects (d. Fig. :3, illus­ trating the TyPoFs for a player to be in frout of player 82 as seen fro111 the left. penalt.y spot, and for a player to be in front. of the nght goal area as seen from a view­ point marked with the star). Similar to the system of 'ideal meaning' and 'geometric description functions' in the analysis of Herskovi ts ( cf. [Herskovits, 1986]), we sep­ arate the influence of t.lw objects from the core meaning of the predi cat.e: the assumed particular influence of the argu ment.s, i. e., the objects involved , has to transform the proposed general influence of t he spatial relation by adapting it. t.o t.hc sha.pes, sizes , and possibly also the parts and tlw functions of the objects involved. To cover t.he simi larity between a ll instances of a. spatial relat ion , we assume one function which describes t.he typicality distribution not. - like TyPoFs - wit.h respect. t.o the co­ ordinates of t.he objects, but. wit.h rt>s1wrt. to what. we call essential parameters. The essential parameters, e.g., dis­ tance, angles, and scaling factors, are abstractions from the concrete coordinates. From this function - the Typ­ tcalzty Schema of a spatial relation ( cf. Fig. 4) - , the Ty PoF of any particular instance of that spatial rela­ tion is derived. All these TyPoFs inst.ant.iated from one Typicality Schema differ only with respect to the func­ tions used to calculate t.he essential parameters from the objects' coordinates.

The influence of the objects is encoded by TyPoF In­ stantiation Rules (for short: !-rules) which - in a way ·­ spr·ead the typicality distributions encoded in the Typ­ Icality Schema around the objects in the context, thus developing the appropriate TyPoF. Each 1-rule specifies, according to the object attributes, a set of functions for calculating the essential parameters. These parameter· fun ctions transform the coordinates of the objects, i.e., the information in the percept, to the essential para­ meters needed by the Typicality Schemata. Since, for example, distance, the essential parameter used for 'be­ ing in ', '-.at ', '- close to', '- near' (cf. Fig. 4), but also mvolved 111 'being left of', '- right of', '- in front of', and '-behind', is defined only between zero-dimensional geometrical objects, we have to reduce - or idealize - higher-dimensional objects to points in order to apply the corresponding typicality schemata. To find the dis­ t.ance between two two-dimensional objects, we first. have t.o look for those two points of the objects' borders which are closest to each other: their distance is the distance between the objects, i.e ., they in fact idealize the objects.

If an object. is simultaneously restricted by several propositions, the corresponding Ty PoFs have to be com­ bined appropriately: algebraic average provides a simple method which also allows us to distinguish compatible and incompatible sets of propositions. The TyPoFs of compatible restrictions interfere in such a way that some locations remain with high typicality in the combined field (~ 0.8 , cf. Fig. 5). 3 lncompat.ible restrictions resuit. in a con1hined TyPoF with relatively low maxima -­ reflect ing the fact. that. incompa.t.ible r~·st.rict.ions cannot. all be fulfill ed at. once.

. 3 ln Fig. 5, the combination of the following local restric­ t.Jous for B-6 is depicted: (between W-3 Right Goal), (uear Center Circle), (at Halfway Line) , and addit.ioually ('reach­ able wit.h maximal speed in 3 sec'):

To rest rict. the momentary velocity of an object, the duration of an event., or t.he temporal relation between events, the concept. of TyPoFs was extended: the do­ main of temporal TyPoFs whi<+ are used in the first. stf'p of concretizing obviously iL- only one-dimensional in contrast to t.he TyPoFs controlling location or veloc­ ity. TyPoFs rest ricting t.lw velocity correspond rather directly to TyPoFs restrict.ing the location of an object: the former bind together two 'incarnations' of the ob­ j ect at. consecutive instants . The velocity restriction is interpreted as a rest rict ion of the location at the next moment.. Also in t.hcse cast's, we have t.o differenti­ ate between the influence of t.he velocity predicate and t.he influence of tlw object.: a ball. for example, reaches usually a Inuch higher velocity than a player , a differ­ ence illustrat.cd by Fig.s () and 7: both figures show the typicality distributions for the three qualitative levels of speed considered in A NTLTI'vl A - namely low medium and high - , the first for a player, the second f~r the ball (the typicality distributions for the locations after two seconds are shown). Since it. is more difficult to change the direction of lllot.ion within one t.ime quantum the faster an object. tnows , the 'a.ngl~' ' of t.lw speed-TyPoFs varies wi t.h tlw speed level.

A seqtH'Itce of sets of elementary restrict ions of loca­ tion and s1wf'd, like the propositional elementary struc­ ture analysed fr0111 an event. proposition , is concret.ized by reconstruct.ing a corrC'sponding sequence of static sit­ uations - like a film - incrementally moment after mo­ ment: the last. reconstructed 'snapshot' is used as the context for the concrf't.izing of the next instant by means of combining the velocity rest rictions of the already re­ constructed time point and the locat.ional restrictions of the instant being considered; the positions of the objects at one moment are used as starting positions for the ap­ proximation of their positions one time quantum later (an example is included in the long vPrsion). The speed Tyl'oFs may be usf'd additionally for adequat-ely reducing the range of typical locat ions: those not reachable for the concerned obj ect within one time quantum by maxi­ ma l speed may be optionally 'faded out.' in ANTLIIvlA.' 1

4 Conclusion

With ANTLIMA, t he systematic application of t he ref­ erence semantics approach for combining vision and na t­ ural language systems can be studied : the reconstruction of perceptually absent referents in the form of menta l images is to be used in a list.encr model for anticipa.t.­ ing what the audience can comprehend , with the goal of increasing appropriately the coherence of the generated text. We have demonstrated herein t he basic principles of such a reconstruction (the system is presently real­ ized on CLOS and CLIM); on this foundat ion, further studies, especially concerning the treatment of three­ dimensiona l objects, the concept ua lizat ion of movements as deformable one-dimensional objects called 'traj ecto­ ries' and their use (cf. (Hays, 1989]) , and the re-analysis of the images with the feedback to text planning and generat ion , are currently in progress.