Event Models for Recognition and Natural Language Description of Events in Real-World Image Sequences

1. Introduction

This paper deals with high-level image-sequence intet·pretation, i.e. with obtaining a meaningful description of time- varying visual data at a high level of abstraction. While in the traditional single-image paradigm of computer vision an interpretation in terms of object names and locations may be considered adequate, this is clearly not the case for an image sequence. Image sequences have much the same semantic potential as silent movies. Hence computer vision should ultimately be able to extt·act results comparable to human understanding of silent movies. While the silent-movie paradigm does not define "high-level interpretation". it suggests conceptual structures which at·e similar, if not partially identical, to meaning representation of natural language. As Miller and Johnson-Laird [1] put it: "Percepts and words are merely avenues into and out of this conceptual structure."

In this contribution we propose conceptual structures fat· a limited domain: motion in a traffic scene. Traffic scenes have been used in Hamburg for a long time both for vision and natural language research. On the vision side techniques have been developed to isolate moving objects and obtain 3D-shape descriptions [2]; the natural language group implemented a powerful dialogue system for simulated static scenes [3] . Recently efforts were started to connect both systems, with the ultimate goal of obtaining natural language descriptions of real-world image sequences [4] .

EVENT HODELS FOR RECOGNITION AND NATURAL LANGUAGE DESCRIPTION OF EVENTS IN REAL-HORLD IHAGE SEQUENCES

Bernd Neumann and Hans-Joachim Novak

Fachbereich Informatik, SchluterstraBe 70 0-2000 Hamburg 13 , W. Germany

Our approach to motion representation and recognition is strongly related to the pioneering work of Sadler [5] and the more recent work of Tsotsos [6) . Both use a hiet·at·chy of motion concepts to bui ld a high-level description from low-level information. Motion primitives such as 'location-change' can be immediately retrieved from basic scene analysis data. Sadler also generates simple vet·bal desct·iptions based on instantiated motion concepts corresponding to English verbs of motion and directional advet·bials.

In section 2 a representational scheme will be presented which extends this work. We introduce "event models" which specify certain interesting subspaces of four-dimensional space- time continuum (much in accord with Webster 's definition). Event models support bottom-up scene desct·iption as well as question answering.

In section 3 we discuss issues of evaluating event models.

2. Event models

In the preceding section an event has been loosely introduced as an interesting occur'rence in a time-varying scene within a certain time interval and within cet·tain spatial boundaries. In this section we shall define event models as a representation of classes of events and as a tool to recognize events in a given scene. Event models will be organized around verbs of change - curt·ently restricted to verbs of locomotion - such that each event model describes events whose natut·aJ language description involves a pat·ticular verb.

It is convenient to sidestep scene analysis issues by assuming a certain standard rept·esentation of the analyzed scene. We use a "geometrical scene description" in terms of symbolic object names, object types, and 3D shape and position for equally spaced time slices (see [7] for details).

Event models specify predicates about the geometrical scene description in a relational form, e.g.

(!SA OBJ1 VEHICLE) (BEHIND OBJ1 OBJ2):JlTIME1

The first component of a tupel specifies the relation, all other components are either constants (VEHICLE) at· unbound token variables (OBJ1, OBJ2 , TIME1). The :;)-operator - introduced in [6] - denotes evaluation at a time instance or for all instances in an interval. Time is considered discrete. Relations may be primitive (dir·ectly computable or· r·etr·ievable fr·om the scene data) or defined in terms of other relations.

A fir·st example of an event model is given t)elow. It specifies predicates which must be true for an event to be ver·balized using .. move .. .

(EVENT - MODEL E- MOVE (PARAMETERS OBJ1 TIME1 TIME2) (KERNEL (IN- MOTION OBJ 1)@(TIME 1 TIME2)) (C-FRAME C- MOVE1 C-MOVE2 .. . ))

An event model consists of a name and parameter section similar to a procedure declaration. The kernel specifies predicates with respect to the geometrical scene description. IN-MOTION is considered a primitive pr·edicate. The event model is linked to possible case frames listed in the C-FRAMES statement . A case frame connects variables of an event model with the deep cases of a natural-language utterance. C- MOVE1 involves a source and a goal case, other· case frames may invotve other case combinations.

(CASE- FRAME C-MOVE1 (CASES · (VERB MOVE) (AGENT OBJ 1) (START TIME1) (END TIME2) (SOURCE OBJ1@TIME1) (GOAL OBJ 1 @TIME2)) (E-MODEL E-MOVE))

In the case of question answering the parser uses the variables specified in a case fr·ame to generate additionat constraints supplementing the event kernel such that any instantiation of the event model is a positive instance for the question at hand. From the question

.. Did the red lorry move from· the drive-way to the parking lot? .. we get the constraints (IS A OBJ 1 LORRY) (COLOR OBJ1 RED) (WITHIN (TIME1 TIME2) (PAST-TIME1 PAST-TIME2)) (!SA SOURCE1 DRIVEWAY) (MARK SOURCE 1 OBJ 1 @TIME 1) (!SA GOAL 1 PARKING-LOT) (MARK GOAL 1 OBJ1 @TIME2)

Conversety, if bottom-up scene description is required, case frames specify from which (instantiated) variables of an event verbatizations are to be generated. Many of the involved mechanisms, e.g. the generation of time bounds or MARK for relating two locations are availabte from previous work [8,4], but they must be remodelled to permit both bottom-up and top-down processing.

Event models may be partially defined by other event models, hence a hierarchy is induced. Modell is said to be a specialization of model2, if its kernel contains the kernel of model2 (after uni- fication of variables) or, in a logical interpretation, if it implies model2. In the following example it is assumed that APPROACH and

RECEDE have already been defined, possibly using MOVE. (EVENT-MODEL OVERTAKE (PARAMETERS OBJ1 OBJ2 TIME1 TIME2) (KERNEL (MOVE OBJ1)@(TIME1 TIME2) (MOVE OBJ2)@(TIME 1 TIME2) (BEHIND OBJ1 OBJ2)@TIME1 (BEHIND OBJ2 OBJ 1)@TIME2 (WITHIN (TIME3 TIME4) (TIME1 TIME2)) (BESIDE OBJ1 OBJ2)@(TIME3 TIME4) (APPROACH OBJ 1 OBJ2)@(TIME 1 TIME3) (RECEDE OBJ 1 OBJ2)@(TIME4 TIME2)) (C-FRAMES ... ))

As stated earlier many verbs of locomotion cannot be used without cer·tain knowledge about what is typical and what is not. Consider the German verb .. weitergehen .. (appr·ox ... to carry on walking .. ). On the sur·face it may describe an uninterrupted walking event, but clearly it implies that halting had been expected. Similarly, .. turn off" is used when an object continues its path along a way which deviates fr·om the natur·al extension of its previous path. Expectations may be specified in event models using the following notation:

(EXPECT <expectations> IN-VIEW- OF <premises>)

The EXPECT construct relates two partial scene descriptions, where the description termed <premises> pr·ovides the scene data on which the expectations ar·e to be based. Usually the whole scene up to a certain time or except cer·tain components constitutes the premises. In those cases it is convenient to use a shorthand THIS-SCENE BEFORE <time> or THIS-SCENE WITHOUT <objects>. In the following example we define the event model CARRY-0N-WALK1 using the EXPECT constr·uct.

(EVENT-MODEL CARRY-ON-WALK (PARAMETERS OBJ1 TIME1 TIME2) (KERNEL (WALK OBJ1)@(TIME1 TIME2) (WITHIN TIME3 (TIME1 TIME2)) (EXPECT (STAND OBJ1)@(TIME3 TIME2) IN-VIEW-OF THIS-SCENE BEFORE TIME3)) (C- FRAMES ... ))

The EXPECT notation is a first step towards interfacing event models with several knowledge sources other than scene data, prominently experience about past events and common sense. In our system we plan to generate expectations by matching the premises with encodings of typical events.

3. Event recognition

An event model may be viewed as a template which must match pertinent scene data, verbal information and background knowledge for an event to be tr·ue. From the examples it should be plausible that predicates are involved where the truth value can indeed be determined given certain arguments and the above mentioned knowledge sources. Some of the predicates are ver·y simple (e.g. IN-MOTION), others are less so (e.g. EXPECT). Very tittle has been said, however, about the process of instantiating a complete event model, i.e. event recognition. In this section we shall discuss some features of the control structure. An implementation is currently being prepared.

The genet·al framework is a backtt·acking contt·ol regime induced by alternative predicate instantiations. Time vat·iables, howevet·, play a special part in event recognition and are treated differently. Note that many predicates involving time as well as durative events like MOVE are also satisfied for subintervals if they at·e satisfied fat· some interval at all. Fot· these cases it is convenient to keep track of possible values using linear inequalities. Following the idea in (9]. we can then apply linear programming methods to determine satisfiability each time a new constt·aint is added. Solutions may be obtained satisfying the inequalities and maximizing some criterion function, e.g. ''the most recent instance·· or ''the longest interval"'.

For an effective seat·ch it is necessary to contt·ol the order of alternative instantiations, the order of predicate evaluation and the ordet· of invoking event models. A useful criterion for controlling instantiations is temporal and spatial pt·oximity to the current focus of attention.

The order in which the pt·edicates of an event model should be evaluated depends decisively on the constraints imposed by the natural language interface. Decision questions may propose specific events (e.g. "'Did a yellow car tut·n off Schlueterstt·eet?'') where event model instantiation is constrained by agent and source specifications. On the other hand, event models must also be instantiated in a free verbalization context (e.g. "'What happened?"'). It is one of the advantages of the strictly declarative definition of event models that they can be employed for all cases ranging from recognition to verification. For an effective evaluation predicates may be dynamically ordered according to their degree of instantiation such that the branching rate can be kept low.

Finally, for bottom-up scene description one has to hypothesize one out of many event models. We pt·esently consider ca. 70 verbs of locomotion (listed in (10]) and a corresponding number of event models. They are organized into a specialization hierarchy as discussed in the pt·evious section. This makes it possible to proceed from general event hypotheses (e.g. MOVE) to increasingly special event hypotheses (e.g. CROSS) similar to the control structure used in (5], for that mattet·, in othet· hierarchical systems.

4. Conclusions

We have proposed a representation for event models which is designed to meet four main objectives.

• (i) Event models provide a precise yet readable definition of a class of events. This is achieved using a relational notation with a clear logical interpt·etation.
• (ii) Event models support event recognition in the framewm·k of computer vision. A scene is assumed to be analyzed up to the level of object recognition before event recognition begins. Although the state of the art does not yet permit automatic object recognition in real-life traffic scenes, this seems to be an attainable goal.
• (iii) Event models are linked to a corresponding natural language desct·iption. Although theit· conceptual structure is oriented towards visual data, they suppot·t both question-answet·ing and free verbalization.
• (iv) Event models may t·efet· to expectations and hence interface to contextual knowledge , experience and common sense. This allows to model events in terms of deviations from the expected - a feature whose importance has long IJeen t·ecognized (11].

The evaluation strategy has been outlined. An implementation is undet·way.