Selective Attention in Dynamic Vision

1 Introduction

One important use of high-level vision is to guide the reasoning of the whole vision system, acting as an atten­ tional mechanism that exploits knowledge of the specific problem being solved. Here we develop a general frame­ work for such an attentional mechanism and its applica­ tion to understanding dynamic scenes.

Although the example used in this work is taken from the application domain of road traffic, this work is con­ cerned with the general vision problem of understanding what we see. The case being considered here is a fixed official-observer 1 (camera) watching a dynamic outdoor scene for which a low-level vision system has been built that returns 3D pose positions of those dynamic objects visible to the camera. This processing is made feasible by having a known scene layout and expected object types. Further prior knowledge is used by the higher-level rea­ soning system which includes more extensive data about the spatial-layout and typical object behaviour [6] .

In this approach, we also identify some key features to bias the attention of the running system so that it searches for these features, identifying any occurrences. Often when we interact with the world it is with some purpose in mind. This purpose enables us to grade the available features and select the essential properties from the various sensory inputs. Without such selectivity we would be overwhelmed by the available data. In this paper we show how task bias can be used to make the reasoning performed relevant to the task at hand.

·This work has been funded by SERC under a CASE award with the GEC Marconi Research Centre.

1 The name official-observer denotes the viewpoint of the system, i.e. the camera's field-of-view.

Hilary Buxton

Cognitive & Computing Sciences University of Sussex, Falmer, Brighton BNl 9RH, UK

2 Issues

The starting point for this work is a temporally ordered stream of 3D pose updates representing the movements of dynamic scene objects in a known scene. The two key problems encountered when processing this data are: (1) The computational load of determining what all the ob­ jects are doing. For instance, we may not want to con­ sider all object interactions. (2) Even in the relatively compact representation which comprises the set of pose positions there is the problem of managing the tempo­ rally evolving mass of input data.

A fundamental step in the approach taken here sepa­ rates the simple and complex operations that can be per­ formed upon the input data, and uses the results from the simple operations to guide the application of appro­ priate complex functions. The goal here is to reduce the irrelevant application of the more computationally expensive complex operations by making the reasoning performed relevant to the task at hand. In the context of understanding what a scene object is doing, simple op­ erations include those that return values for positions, velocity, nearest-object, while complex operations per­ form computations such as path-prediction and other­ object-location. This decomposition of the computation is also combined in this approach with the simplifica­ tions provided by the deictic representation outlined in the next subsection and with the agent based formal­ ism using Bayesian network updating to give a general framework for implementation.

2.I Object representation

Related to the issue of selective attention by the official­ observer is set of properties associated with the object being observered. Gibson [4] called these properties "af­ fordances". Here we are trying to describe the behaviour of dynamic scene objects. Generally, these objects have an intrinsic front so that when we describe the actions of such objects, we use the object's own local coordinate system. In effect, we have translated our own frame of reference to that of the object. This is one property of the deictic representation 2 used to describe the activi­ ties of the scene objects. Other properties are used to simplify the "typical-object-model" 3 that is used to se­ lect which data gathering "visual-operators" 4 should be activated. There are two classes of visual-operator used here. Peripheral visual-operators are used to perform simple computations upon all pertinent data, and atten­ tional visual-operators are used to perform specific local data-gathering computations. The typical-object-model also determines which values, if any, should be retained (latched) for the next clock tick. During each clock cy­ cle, each "foveated object" (called an "agent") uses it 's typical-object-model to select which attentional visual­ operators to run on the next clock-tick. The results from these selected visual-operators are fed back to the appro­ priate agent providing task related features upon which to base it's next selection .

2 For details concerning deictic representation see [2; 11].

The central premise behind this agent selective view­ point is that an agent need not name and describe every object in the domain, but instead should register infor­ mation only about objects that are relevant to the task at hand. At any one moment the agent 's internal repre­ sentation should register only the features of a few key objects and ignore the rest .

The system is currently set up with just three agents. 5 The egocentric data gathering of each agent is integrated by the official-observer. This is simplified by the fact that the official-observer originally selected which ob­ jects to attend and thus the execution of the typical­ object-model is like the official-observer considering the possible next actions of an object it is attending.

2.2 Selection task

The typical-object-model run by each agent operates at the wrong computational level to act as an attentional controller for the official-observer. This guiding task should be based upon both ( 1) the peripheral visual­ operator outputs, and (2) the results from each of the agents. These outputs correspond to the simple and complex operations mentioned above. We can sum­ marise this by considering a general case:

Definition 1 Given two operators OPl and OP2 where {1) OP1 is much less complex and runs faster than OP2, {2) OPl does not return an answer that is as useful as OP2, (3) OPl is true in all situations where OP2 would provide a useful result (4) OPl is false in all situations where the preconditions of OP2 would not be met or where we do not want to apply OP2. If these conditions are fulfilled, OP1 can act as a guide for the application ofOP2.

When considered as part of the system, the general case is complicated by (1) the data-association-problem 6 at the observer-level, (2) the selection of which OP2s to run is dependent upon factors other than the OPl re­ sults alone, for example, later in this paper we will de­ scribe how observer-level results are used to influence the allocation of agents.

3 Denotes the concept of representing the "typical" be­ haviour exhibited by the class of object being observed. In effect a set of rules for mapping a given "situation and recent history" in the form of key features to effector actions.

4 As used by Agre and Chapman [1].

5 Pylyshyn and Storm [10] present psychophysical evidence that there is a numerical limit of four or five on the number of objects that can be tracked at one time.

6 The problem of binding related data together.

3 The basic observer-level

In this section we have chosen functions for OPl and OP2 that reflect the example used in section 4. These functions are used for spatial reasoning about the visual data. One of the most elementary spatial primitives in spatial reasoning concerns the relative nearness of one object to another. In the example below we use a prox­ imity measure as our simple operator using the assump­ tion that objects generally interact with those that are nearby. The nearness measure is based on a distance metric scaled to give an exponentially higher value to nearer objects and discretised to the integer range (0 , ... , 5] (this is also the range used by N;(t) in table 1) . The more complex operation takes this result and determines the bearing of the nearest (other) object relative to the object's frame-of-reference. The result is a deictic refer­ ence such as behind-me , beside-me or infront-of-me. (Note that conditions {1,2,3,4) are fulfilled- the com­ plex operator only works when given the result saying which, if any, is the nearest object.)

We are using Bayesian networks [9] which combine an inference mechanism for uncertainty together with an efficient propagation of evidence and expectations. Recently, a few researchers have used Bayesian networks in dynamic domains [3; 8] where the focus is reasoning over time. This is not the only focus here, we also use another Bayesian network to assess the task relatedn ess of the actions selected so far , providing a task-direct ed attentional mechanism.

3.1 Selecting interesting features

To combine the proximity information that develops over time we have used a dynamic form of Bayesian network (DBN). That is, the network evolves structurally over time to capture the changing proximity relationships be­ tween scene objects. In the case outlined here, the graph formed describes which object is near another one. The node values hold data that contributes towards a mea­ sure reflecting how interesting this relationship is to the official-observer watching the scene. The basic graph primitive is depicted as:

This graph describes a mutual proximity relationship be­ tween objects A and B, capturing the primitive notion of object A being near B and B being near A. The relationship node R denotes this mutual proximity and holds a belief value that reflects how interesting this is . The states and purpose of the two node types are given in table 1. The inputs to DBN are taken from the peripheral visual-operator outputs (called "peripheral­ aspects") . These outputs are given once every system­ cycle, which means that the values held at time t are from the current outputs. At the end of each system­ cycle, the storage is "moved" (by pointer reallocation)

so that what was held at t is now held at t - 1. When the peripheral-aspects become available, they are used to re-construct the DBN.

The DBN is composed of temporally separated sub­ graphs that are interconnected. DBN nodes are ac­ cessed via thei r respective subgraph, providing an im­ plicit temporal reference relative to now. Each DBN subgraph has vertices (V) and edges that are denoted by edge-~esti~ations (ED) and edge-sources (ES) . Denoting edges m th1s way enables us to describe the links between subgraphs. In the example here, we will use only two subgraphs , however, a general definition is: DBN = (TS, {SG1, .. . , SGn}) where TS is a list of subgraph indices ordered to r eflect their current temporal status. SG; = (V;, ES;, ED;) where V; are SG;'s vertices, ES; are SG; 's edge-sources and ED; are SG; 's edge-destinations. The edge names are defined so that in addition to refer­ ring to the vertex-index they also refer to the rela­ ~ ive time, e.g. edge-index = ( deict ic-time- index, vertex­ mdex) where deict ic-time- index E {prev, now, next}. 7 The construction of the DBN graph consists of two stages, node-building and node-linking, which are described with algorithms in the longer version of this paper [7] .

Here we are using a DBN graph structure with two types of node, N and R, and with directed edges that describe one of two types of relationship, spatial or tem­ poral. The directed edges hold the fixed conditional probabilities given in table 2. The spatial relationship is specified by the matrix MRIN, capturing the belief that a proximity relationship becomes increasingly more in­ teresting as two objects are identified as being nearer to each other. Figure 1(a) shows that when we evolve the network over time we make use of maintained official­ observer object references to the image data. If, at the current time point , A and B are still near to each other we can form temporal links from the previous node to the current ones as shown in figure 1 (b) . From the rel­ ative temporal position of the previous time point the temporal links are to the next time point . These di­ rected temporal edges each hold one of the matrices rep­ resentin~ t.emporal continuity, here this is either MNIN for prox1m1ty, or MRIR for the pairwise relationship. Fig­ ure 1 (c) shows how we extract a tree from the D BN graph which is done for the parsimonious reason that a tree is easier to evaluate than a multiply connected network.

7 We also use t to refer to now, t - 1 for prev and t+ 1 for next.

Figure 3: The roundabout , cam­ era position and typical traffic flow.

This graph simplification preserves the results obtained from .the previo~s value propagation allowing them to contnbute effectively to the formation of the new belief about the relationship that now holds between object A and B. In the exam pie developed in this paper each tree produced only holds values from the current and previ­ ous time points. If the DBN graph contains more than one relationship , then a tree is formed for each relation­ ship. Once all propagations have completed the most interesting relationship can be selected.

The DBN network changes structurally over time to reflect the relationships between the objects of interest . For example, figure 2 shows what happens if N8 is miss­ ing for two time points ( t3 and t4) due, say, to occlu­ sion and the nearest neighbour to N A now becomes N c. When NB is detected again it is identified as being near­ est and the new links and nodes in the network are cre­ ated to reflect this.

The DBN itself does not hold an overall picture of what is happening. For the creation of a fuller picture we use a separate Bayesian network called TASKNET.

3.2 Meta level

The DBN provides likelihood measures of which pa1rs of markers, if any, are worth further attention. Fur­ ther attention involves the allocation of an agent, a lim­ Ited resource, that we want applied to the objects that are the most relevant to the current official-observer tas~. To do this we use the results from the agent level, wh1ch are composed according to their DBN relation­ ship. The resulting composite is input to the pair's ded­ icated TASKNET which builds a coherent interpretation of the temporally evolving object relationship. Each marker pair (i,j) , has a distinct TASKNET which we can denote TASKNET(i ,j)· The TASKNETs have a tem­ porally fixed tree structure with conditional probabilities that are defined before runtime to reflect a preferential bias towards a feature that is deemed to be most interesting. The input nodes represent key features rel­ evant to the task the TASKNET has been constructed to identify. The output root node represents the overall belief, based on the evidence collected so far, in a set of candidates consisting of the wanted task, related but unwanted tasks, and the default unknown task.

Allocating an attentional process does the following: (1) ensure that a TASKNET is collecting information related to the pair, (2) ensure that an agent is running on both selected objects. When an uninteresting situa­ tion is recognised, we terminate attention by instantiat­ ing the value held in the pair's current DBN reference node. On the next clock tick when the DBN is updated and the inference algorithm run, if the relationship is still present, a high ignore value placed in the reference node at what is now t-1 will ensure that, unless a po­ tentially interesting indicator is present at time t, the relationship will be ignored. This high ignore value is temporally propagated . Using this method, once a re­ lationship is identified it can be ignored or continue to be watched. This attentional system has three proper­ ties that are used to guide the reasoning of the runtime system. ( 1) Focus-of-attention - only the selected set of target hypothesis and their associated functions are updated. (2) Terminate-attention - once a target hy­ pothesis has been confirmed or denied to an acceptable degree of confidence we can stop all activities associated with it . (3) Selective-attention- we can dynamically se­ lect the current most interesting hypothesis to watch .

4 Road traffic example

The overall goal of the project is the formation of con­ ceptual descriptions that capture what is happening in the scene. We would like this conceptual description to evolve in conjunction with the scene events and not be dependent upon the observation of a whole episode before saying what happened. The uses for this work in­ clude identification of unusual or illegal traffic behaviour where the system could be set up by the road side and used to select when to record relevant passages of video footage for subsequent analysis. Such intelligent surveil­ lance of a scene must be capable of recognising the sit­ uated actions of the subjects being watched and mea­ suring these observations against some model of correct behaviour. Although a surveillance system does not ac­ tively participate in the world, it still requires models of perception and action . These models need to incor­ porate suitable representations of space and time. The application domain of road traffic surveillance is selected on the pragmatic grounds of having access to data col­ lected at a German roundabout and processed by the VIEWS project vision system (some details of which are given in [14]).

The results from the low-level vision system take the form of 3D pose positions which can be transformed to provide a ground plane projection. This makes it easier to reason about vehicle interactions, position and shape. Figure 3 shows the camera's field-of-view and, although less obvious, the camera position does affect the contents of the frame updates because we are dependent upon what is visible from the camera position not what is visible from the overhead view.

The purpose of this example is to show that the DBN and TASKNETs can pick out a pair of vehicles that are performing an overtaking episode. This "overtak­ ing" is likely to be non-intentional behaviour which just depends upon how the vehicles on the road are organ­ ised (and contributes to the uncertainty of identifica­ tion) . We can model a !-am-being-overtaken episode from five temporally ordered deictic event primitives [behind-me] [behind-and-beside-me] [beside-me] [beside-and-infront-of-me] [infront-of-me] . The conjunction of deictic references that point to the same thing is needed because we are not dealing with a sub­ ject that is a spatial point - a vehicle has spatial extent, a front, sides, and rear [5].

The DBN network will be using the proximity mea­ sure described earlier together with three conditional probability matrices shown in table 2, which are for the node types given in table 1. These matrices were derived from careful consideration of the possible in­ put values. The results from the DBN (marker pairs of the form ( i, j)) are used to select which agents to run on which markers. Agents produce two types of output, ''effectors" and "agent-events" . Effector output is used to select which attentional visual-operators to run on the next clock tick, and agent-event output is used as TASKNET input. Each agent-event is made of two components, the marker reference of the near­ est object and the deictic event primitive E {behind-me, beside-me, infront-of-me, blocking-me} . The input given to TASKNET(i,j) is derived from the composition (performed according to the rules given in table 3) of the agent-events provided by the agents running on markers i and j . In this example the TASKNETs have a very simple structure consisting of two nodes, one of each type given in table 4. CE(i ,j) is the input node, LE(i ,j) is the root/output node, and they are related according to the conditional probabilities given in table 5. The re­ sult from the agent-event composition is given as an evi­ dence node to their TASKNET. The result of running the Bayesian inference algorithm is used to guide agent allo­ cation as described earlier by identifying the likelihood that the relationship is either overtaking, following, queueing or unknown.

The high-level vision system described here has been implemented in Lisp. The results from the frame se­ quence in figure 4 are both incorporated in the figure and shown in table 6. The filled in vehicle shapes are uninteresting peripheral objects, the outlined shapes are selected objects, and the number near each vehicle is it's marker reference. The sequence begins with two vehi­ cles being selected because they are near each other. At frame 24, they are identified as following one another and are ignored. In frame 48, the pair (3,2) are selected because they were near each other in frame 36. In frames 48 to 72 the pair (3,2) are not mutually close together. At frame 84, possible overtaking or following is identified and identifying overtaking is given preference since it is the task objective. During frames 96 and 108 one of the vehicles occludes the other from the camera. By frame 132 overtaking is positively identified.

5 Discussion and Conclusion

Many of the aspects of the attentional control system outlined in this paper have been influenced by previous work. The use of key features and markers is based on work by Agre and Chapman [1] although it was neces­ sary to extend their techniques to include multi-agent viewpoints which contribute to the official-observer view for surveillance. The imposition of the local viewpoints in the deictic representation of spatial relationships also simplifies and decomposes the possible interactions be­ tween movin_g objects to allow an efficient response. Dean et al. l3] first proposed DBNs using clique trees and a fixed , temporally-repeated tree structure but here a more general, dynamic restructuring is required to model the set of interactions. Extracting the relevant subgraph (tree or singly-connected network) by a priori identification of older edges which can be removed seems novel and extending this to more complex DBN graphs is challenging! The use of a task-based Bayesian network is mentioned in Rimey and Brown [12] where it is used to actively direct the camera using geometric relationships. Here, however , our application requires scene surveil­ lance based on spatial (and temporal) reasoning relative to a static camera. The combination of dynamic net­ work propagation with task-related evaluation promises a highly effective attentional control mechanism which can be distributed under the agent formalism to deliver real-time performance although we are interested in the competence at the moment .

In particular , the results reported in section 4 show that the approach works well for simple tasks like mon­ itoring overtaking episodes in the roundabout sequence. Future work will investigate (1) how well it copes with more complex, multi-object interactions and (2) whether the task-level can be extended to recognise temporal se­ quences of tasks which have a plan-like structure. In addition, instead of just starting from the precomputed pose positions, it would be more effective to experiment with a running vision system that was actively interfaced to the visual-operator primitives. The attentional mech­ anism could then help reduce the visual search involved in computing the pose information [14]. Visual search is a notoriously complex problem [13] which would benefit from guidance that exploits knowledge of the task to be performed.