Qualitative Recognition of Ongoing Human Action Sequences

1 Introduction

Social animals, including humans , have high compe­ tence for observing, understanding and reacting to other agents' actions effectively. Action understanding is in­ dispensable for intelligent behavior in mult.i-agent envi­ ronments. Also, the coincidence of higher intelligence and advanced action understanding ahilit.y suggests that action understanding may be deeply related to the fun­ damentals of natural intelligence.

Intensive research into this issue has been done in AI, especially concerning the plan-based understand­ ing of natural language [Cohen and Perrault, 1979; Allen and Perrault., 1980; Litman and Allen, 19R7; Wilensky, 1983]. However, not much is known about. its prerequisite; the more basic form of action understand­ ing: qualitative recognition of physical action sequences based on visual observation.

Early attempts at this problem dealt. with extracting symbolic descriptions from motion pictures of simple two dimensional figures moving around [Tsuji et a/., 1977; Thibadeau, 1986]. Previous vision research in "motion understanding" has focused on extracting three dimen­ sional motion parameters from image sequences . But this bottom-up approach has not reached the stage of providing the semantic information we need. Recently, an "assembly plan from observation" system [Ikeuchi and Suehiro, 1992] was developed which extracted fine motion sequences from transitions among face contact states acquired by a range sensor . But this theory does not account for the continuous nature of human action sequences.

Our research goal is to provide a framework for tem­ poral segmentation and qualitative classification of con­ tinuous human action streams carried out in a three di­ mensional world. It. is intended to be suitable for real time processing. \Ve also propose this as a cognitive ar­ chitecture for such tasks .

The power of active and purposive strategies in vision has been widely rewgnized [Bajcsy, 1988; Aloimonos and Weiss, 1988; Rimey and Brown , 1992]. They are es­ pecially important when the observer is involved in ac­ tion [Ballard , 1989; Whitehead and Ballard, 1990] . Al­ though we do not. go into the details of vision processing in this paper, we show that active selection of input in­ formation is necessary in the temporal domain as well as in the spatial domain. Also, using attention control for both visual processing and internal representations, we can establish symbol grounding in a dynamic envi­ ronment.

We start the body of this paper by deriving fundamen­ tal methods of action segmentation and action classifica­ tion from an analysis of the notion of action in section 2. Based on these considerations, we propose an overall algorithm for action recognition in section 3. We also present an implementation of an action recognizer for block assembly tasks in section 4. Then we demonstrate the validity of our methods by experimental results of near real time recognition of several simple human tasks in section 5.

1.1 The Example Task

We choose the following example task both for explaining the concepts described in this paper and for implement­ ing a working system:

Input: Observation, by stereo vision, of a human worker constructing various structures with blocks.

Output: Symbolic assembly plans. Each plan is a sequence of assembly operators with intermediate state descriptions . These are to be used by a robot for reproducing the performance of the task [Ku­ niyoshi et a/., 1989] .

• Assumptions: Explicit information on temporal seg­ mentations and a priori knowledge about. a specific task are not given . Classification of possible assem­ bly operations is given . The worker uses only one hand and grasps only one block at a time . One structure is assembled in one task . Each task is carried out only once from start to finish without a pause. Spatial segmentation is tractable. Com­ plete occlusion of blocks does not occur. There are no "messed up" situations, such as col lapses. Also, blocks are brighter than the background .

2 The Fundamentals of Qualitative Action Recognition

2.1 Extracting the Structure of Physical Actions

One important concept is that of qualitative classes of ac­ tions. We share some basic perspectives on movements and actions with Israel et al. [1991]. First, we want to generate a qualitative description of human action se­ quences that can be used for reproducing the observed task in a different environment. In Israel's words , we are interested in repeatable action classes, not in accu­ rate measurement of bodily movements. Let us consider the example of picking up a block from a table . In every instance of such an action, the hand goes roughly down and up , but it. never takes exact ly the same trajectory twice. It may even take a completely different path when there are obstacles in the way. Second, there are nested action classes which form layered contexts. For example, we are usually focusing on a part of the worker's body while ignoring the movements of the rest, which serve as the background condition (Israel's notion of movement context) . In a block assembly task , when fingers operate on a block in order to align it with another block (Fig­ ure 1), an observer is likely to focus on the fingers and the orientation of the manipulated block, provided that the hand is staying roughly in the same position. When the hand suddenly starts off to. fetch another block, such fine movements no longer are significant and so the ob­ server must quickly switch its attention and follow the hand .

Another important concept is that of actions as causal processes. As Dretske [1988) emphasized, the behaviorof a system (performer) is not merely a sequence of bodily movements nor physical events, but a causal process, a structure wherein internal causes (conditions or events) produce certain external movements or changes. The process begins with the causes and ends with the ex­ ternal events, and comprises time and space sufficient ly large to encompass its constituents. Dretske did not make a clear distinction between a performer's bodily movement and its effect on the environment. But we need more detailed analysis of this, since the only ob­ servable portion of an action is the external part of the process; the movement causing some effect.

Clearly, for action classification , an observer needs in­ formation of not only the movement but also of the ef­ fect, which must be causally linked. For example, picking up a block (operation: PICK), putting it. down (operation: PLACE) , and more tricky cases (operation: N00 I') (where an empty hand (or a hand holding a block) approaches the block (or the table) but leaves without grasping (releasing) the block), all involve qualitatively very similar hand movements (approach and then depart from the table), and they cannot be classified using only movement measurements' .

Da.c;ed on the above considerations, we propose the fol­ lowing principles of action observation and recognition :

1. Observing the performer's bodily movement is nec­ essary but not sufficient . A recognizer must also look for causally lin ked effects in the performer 's surroundings and relate movement. and effects.
2. It is very important to accurately determine the spatio-temporal extent of an action being observed. Otherwise actions will be mis-recognized, because different selections of spatio-temporal extents affect which physical events are selected to be related with the performer's movements.
3. Representations of qualitative movements should be organized so as to reflect the structure of move­ ment contexts. One solution is a hierarchy defined in terms of an inclusion relation between spatio­ temporal extents of the movements 2 .

Let us define three concepts for describing actions from an observer's point. of view: movement , motion , and oper­ ation . A movement takes two arguments base (the mov­ ing/changing thing) and t (a time point), and qualita­ tively describes how base is moving (position, orienta­ tion, and their time derivatives) at t .

We distinguish between movements and motions. A motion of an object (base) is not just moving in some way at. one time instant . but moving toward, or away from, or somehow relative to, some other object(s) (targets) within a given temporal interval T . So, a motion takes three arguments, base , targets, T and qualitatively de­ scribes how base is moving with regard to targets within T.

An operation , a synonym for physical action, takes three arguments base, targets, and T, and is defined as a causal process which begins with a motion and ends with an effect. An effect describes a qualitative change involving base and targets within T .

2.2 Temporal Segmentation and Action Classification

Newtson et al. [1977) carried out a series of psychological experiments which provided a strong evidence that hu­ mans perceive an action stream as a sequence of clearly segmented "action units" , and that the most of the use­ ful information for action recognition is extracted at the temporal segmentation points. Further, he suggested that. the key information useful for action identification is not the individual states themselves hut the transi­ tions from one state to another. Unfortunately, he did not explain clearly how the segmentation is done, nor did he suggest any candidate feature types which clas­ sify actions .

1 Finger movements in grasping and releasing the block are typically very small and therefore almost impossible to detect reliably using vision.

2 For physical movement.s, the inclusion relation between temporal extents coincides with the inclusion relation be­ tween spatial volumes swept by the moving object .

Based on a logico-linguist.ic analysis of actions, Thibadeau [1986] argued that segmentation points should be defined in terms of "second order changes" of interpreted images. He also claimed that people do action classification by checking whether a specific con­ dition holds at the end point of the action. His overall theory is based on the assumption that complete sym­ bolic interpretation of every image is provided for the whole temporal sequence, and the first and the second order changes between the descriptions are constantly computed. This assumption is infeasible for recogniz­ ing human actions carried out in the three dimensional world , especially when using vision modality. We need direct methods to detect segmentation points and to clas­ sify actions . The only direct feature for action segmen­ tation he mentioned is acceleration of a moving object. Unfortunately, we have found that using only accelera­ tion is not a. very effective method for segmentation of real actions.

Our approach is to derive a concrete definition of both action segmentation and action classification simult.a.ne­ ously based on the definitions of physical actions and motions.

Let us assume that an observer sees a physically closed world W containing a fixed set of objects 0 . We de­ fine a state S(t) of W ac; a set of n-ary measurements (for all possible n) of 0 at time t, for exar11ple , posi­ tion/orientation parameters, distances or contact states between pairs of objects. Let P stand for the set of all possible temporal evolutions of S(t). Then, qualitative recognit ion of an action requires cutting a. portion from P and identifying it as one of the known action concepts.

Following the formulation of the notion of concepts by Hobbs [1985], we define a set of conceptual action classes C as a quotient set of P by a. certain equivalence relation Rc: C = P/ R0 · A set of observable qualitative features F is also defined as a. quotient. set of P by an equivalence relation R0 : F = P/ R0 .

We state that. F hac; indexicality to C if and only if the observation process , R 0 , articulates P in the very same way as the concept. formation process Rc does.

An action is a process in which one object affects the state of others, as stated in section 2.1. So, conceptually, an action can be represented using a. tuple (o, T, Sf, SC), where S I is a. subset of the temporally invariant portion of P , SC is a subset of the temporally changing portion of P, o is a subset of 0 which support Sf and SC, and T is a temporal interval Then we can divide Rc into three subrela.tions and relate to Ro, as described below.

Attention: Focused spatial extents o which specify objects, geometrical features or locations , and the tem­ poral extent T. The observer must keep track of o within T . The spatial entities, as they generally participate in motion, will be called: base, targets, and special.

Context: Invariant temporal property of mot ion, SI:

Qualitative movement and relations to other fo­ cused entities (that is, the position, contact state, etc. of the base with regard to the targets).

Change: Qualitative change of relations (held, af­ fixed) or relative position/orientation (on, copla­ nar) among focused entities, SC. This introduces an equivalence relation among possible temporal changes in IV . We hypothesize that, provided that the vision process correctly captures the spatio­ temporal extent which corresponds to the concep­ tual attention and context, there exists an "indexi­ cal feature set" F which can be detected by a very simple visual process. (Examples will be given in section 3.2.) In other words, we hypothesize that action recognition is easy if attention is properly controlled and context is properly maintained.

There is an inherent. difficulty in the observation process: The attention, o and T, are variables which must be filled in by visual observation. In general, the only constraint on their values are imposed by S I and SC. However, S I and SC cannot. be observed until o is determined lwcause they are defined over o. In order to break this circular problem, we take a following method : Assume that at least one object in o is known, call this subset base (it. is maintained by tracking). Then choose a portion of Sf which is defined using base . Using visual search (section 3.2) , find other objects in 0 relative to base in terms of S I. Call them targets and include them in o. The base is reinitialized when the level of hierarchical attention (section 3.2) is changed .

The temporal extent T is defined when the o and SI are determined. In other words, temporal segmentation is done on detection of changes in which object is mov­ ing, whet her it is moving toward or away from an object, and which such ohject., how it is moving, whether it is touching another object., etc. We will describe an effec­ tive method for segmentation in section 3.

2.3 Looking for Relevant Effects

An action recognizer must. find a qualitative effect (change or inva riance) which is causally related with the performer's motion . Selecting the relevant effect is of vital importance t.o correct recognit ion of an action. In a.n everyday world, many things change continuously due to natural phenomena or because multiple agents do different things simultaneously. Relating a change to the wrong agent's motion results in a recognition fail­ me . There are also maintenance actions, actions trying not to change the current state of certain things . If a perfect recognizer were to try to handle both ordinary and maintenance actions, it might. first generate a com­ plete description of the world and then extract all causal relations between motions and changes or inva.riances . In principle this would work, but this would require de­ scribing everything in the world at every instant of time and examining every possible relations among them - whi ch would clearly overload any system.

This is the frame problem [Pylyshyn, 1987] and its general solution is yet unknown. But humans (and many other animals) seem t.o have little difficulty in recognizing other agents' physical actions in real time.

We propose a straightforward hut incomplet.P. solut.ion consisting of two methods:

1. Employ heuristically driven attention control both in the temporal and spatial dimensions to actively find and collect only relevant information.
2. Use predictions based on the routineness of ordinary action sequences.

Our heuristics for guiding the observer's attention are based on the nature of causality. Although causality in general has a very complex structure, we restrict our discussion to a v~ry limited case, the block assembly task. Let us define simple causality in block assembly operations: Let M be a motion of a base concerning targets over a temporal interval T 111 = [tt 1 , t1 1 ], and let E denote a change or an invariant in the physical state of targets over TE = [tf;, tf]. Then a causality M(base, targets, TM) "" E(tare;ets, TE) holds iff TE C T 111 and Touch( base, targets , T 1 ), TT c TE' where TT = [tf, ti]. The condition part is based on the fact that all the objects in the environment (except for the hand) are passive and no mess (dropping or crashing) is al­ lowed, which means that. effects relevant to the current base (hand) movement occur only while the base (hand) touches the targets .

It follows from the above definition that if the tar­ gets are defined and if the temporal segmentation is cor­ rectly done, then E is obtained hy differentiat.in? two snapshots of targets at the segmentation points t f' and t~ 1 . This means that temporal segmentations must he done correctly in real time. Moreover, an observer must predict the location of the targets before the base actu­ ally touches it, otherwise it cannot direct. attention to the relevant region . The prediction is realized by visual search (section 3.2), which directly searches through the 3D space by guiding visual attention.

In addition, the actual location to take snapshots de­ pends on the context. of the action sequence. For exam­ ple , when a hand approaches a block from above, if the observer knows that the performer did a PICK operation before (and so is holding a block), the observer should anticipate a PLACE operation and watch the location just above the target block.

3 The Algorithm for Action Recognition

Based on the analysis given so far, we propose an algo­ rithm for action recognition. First, we describe the main process and then explain briefly about the at.tention con­ trol.

3.1 The Main Process

The algorithm of the main process consists of nested loops executed in parallel. A single loop incorporates the following steps:

1. Do Pre-segmentation: Continuously check the qualitative properties of the movement of the base until one of the anticipated types of changes is de­ tected .
2. Invoke Attention Control: Select a spatial at­ tention control mode (pre-attentive movement de­ tection, hand/object tracking, visual search, etc.) which is appropriate for the action context and in­ voke it.
3. Analyze the Event: Wait until one of the antic­ ipated types of events is detected. On receiving a segmentation event, execute the steps below, other­ wise, ignore it and go back to the pre-segmentation step. For example, when visual search succeeds, the goal location is saved and a segmentation event is raised.
4. (a) Detect a Change: Take a subimage of there­ gion including the target and store it in the en­ vironment model. Compare it with the subim­ age taken at the previous segmentation point of this recognition level.
5. (b) Classify an Action: Select one operation type for the current motion based on the quali­ tative change between the subimage snapshots. Store it into the action history memory (part of the action model).
6. (c) Calculate/Store States: Based on the rec­ ogni?.ed operation type, calculate the change in geometrical relations among the objects under attention. Store the results in the environment model and save them in the action history.
7. (d) Do Segmentation: Mark the end of the cur­ rent mot. ion. If finer levels of action recogni­ tion loops are currently active, force them to do a segmentation and then terminate. Choose the next motion class , as specified by the event type, and instantiate it. Start. processing on it.
8. (e) Antieipate Next Actions: Based on typi­ cal action sequence patterns , predict potential next motions, including finer level ones. Up­ date the stack (and thereby initiate checking for pre-segmen tat. ion events for those motions).

3.2 Attention Control

In our system, spatial attention control consists of three processes: (1) pre-attentive, (2) fixating or tracking, (3) visual search . Information from the pre-attentive and the tracking mocks facilitate pre-segmentation. Regions of interest for change detection are maintained by the fixation mode .

1. Pre-attentive mode is used for continuously moni­ toring the whole workspace very coarsely. It detects a moving thing very quickly and reports a rough es­ timation of it.s 3D position.
2. Tracking mode is used to fixate a moving object. It continuously reports the current 30 position, veloc­ ity, and acceleration of the target . Fixation watches a specified stationary 3D region. It is used to take a visual snapshot of the area.
3. Visual search mode is invoked when necessary (at pre-segmentation). It places a 3D virtual region on the current spatial attention point. (base), moves the region gradually in a specified direction and contin­ uously checks for a specified visual feature inside

it, until something is found or the travel distance exceeds the preset limit. It. succeeds if anything is found, and reports its position, otherwise it fails. The search direction and t.he visual feature to be checked is computed based on the movement of t.he base, and expectations about the next action.

The system also uses temporal attention control. This triggers and stops various modes of spatial attention control based on action (pre- )segmentation. To be specific, tracking is invoked when a moving object is found, subimage snapshots are triggered at segmentation points, and visual search is invoked at. pre-segmentation points.

In addition to spatial attention control and tempo­ ral attention control, we use hierarchical a.t.tention con­ trol. Our recognizer maintains superimposed multiple coarse-to-fine spatial attention regions. The levels corre­ spond to the motion hierarchy which defines dependen­ cies among motion classes (movement contexts) . The overlapping attentions are processed in parallel.

Let us see two examples of indexical ft>at.ures defined in section 2.2.

First .. classification of PICK, PLACE, and NO-OP is done by subtracting silhouette sub-images around the lo­ cation of the targets (the block to be picked up , or the location in which a block is to be put.) taken at segnwnt.a­ tion points. Assuming that. the manipulated block has a brighter color than its hac kground. the three quali tat.ive difference values, decreased, increased, and no-change, directly indicate the actions PICK, PLACE, NO-OP, re­ spectively.

Second example is ALIGN action shown in Figure 1, where block B I is manipulated so that the two faces F1 of B1 and F2 of B2 coincide in a common plane. In the context in which the bottom face of B 1 touches the top face of 132, and B1 is moving in direction 1n so that F1 approaches the common plane, we can detect the coplanar state by merely checking for specific types of specific edge junctions, as shown in Figure 1.

4 Implementation of the Action Recognizer

Here we briefly describe an implementation of the ac­ tion recognizer . Details are presentc>d elsewhere [Ku­ niyoshi et a/., 1992]. It is written in an object-oriented Lisp [Matsui and Inaba, 1990], and wor.ks by message passing. The overall structure of the recognizer is given in Figure 2, illustrating the state of the system while recognizing a PLACE and its sub-action, APPROACH, expecting their end points. The recognizer consists of vi­ sual feature detectors, an action model, an environment model, and an attention stack. The knowledge compris­ ing two models are implemented as collections of nodes. Only the part of the action model and the environment model which is pointed to by the attention stack is acti­ vated at any one time.

4.1 Visual Feature Detectors

The visual routines used in spatial attention control are implemented as separate classes called visual fea­ ture detectors. These are: movement-detector, hand­ tracker , object-finder , change-detector (silhouette differ­ en t.iat.ion). coplanar-det.ector (edge junction classifier). Each detector is connected to an appropriate environ­ ment model node, which has an associated method to control the detector to extract a specific visual feature . Each detector has a specific vision algorithm and main­ tains a stereo pair of visual windows . The visual fea­ ture detectors are executed as parallel background pro­ n·sses using multi-processor vision hardware [Inoue and ivlizoguchi, 1985].

4.2 The Action Model

The motion hierarchy for the assembly task exam­ ple is defined as illustrated in Figure 3. Class "Task" covers the whole spatio-temporal extent of an action­ sequence. There are two types of bac;ic assembly mo­ tions: ( 1) "Transfer", a large movement of the hand in free space, and (2)''Locai!Vlotion", a. hand movement near a target object. "LocaiMotion" is further subdi­ vided into three types of assembly motions: (!)"Ap­ proach", in which the fingers (and the held object, if any) move toward the target objects, (2) ''Depart", sim­ ilarly, and {3) "FineMotion", in which the held object moves in contact. with the target objects.

Each motion class is implemented as a separate node. Each class node specifies segmentation events, and action classification mappings, and has empty slots to be filled during recognition . The slots named "base", "targets", and "special" are for pointers to appropriate environ­ ment model nodes. For example in a PLACE operation, "base" points to a node representing the hand, "special" points to a node for the block held, and the blocks on which that block is to be put are represented by nodes pointed from the "targets". A motion node has vari­ ous other slots. The "initial-state" and "final-state" are for snapshot descriptions of relations among base, tar­ gets, and special. The "operation" is to be filled with a result of action classification . The potential operation types for each motion class are listed in Table 1; this also shows which indexical feature values corresponds to which operations. The mappings are explicit ly repre­ sented in each motion cla<;s node; this information is used by the main recognition process. An instantiated motion class is tagged with an operation type when action clas­ sification succeeds. An example of an instantiated node is shown in Figure 6.

4.3 The Environment Model

An environment model is a. collection of nodes represent­ ing the observer's view of the current task state. It rep­ resents information about focused objects; this is used for the spatial at.t.ention control. The relative positions and dimensions of the base and the target blocks are used to determine the corresponding parameters of attention regions. Whether or not the hand is holding a block is used to help set the snapshot regions appropriately.

The environment model consists of distributed nodes which belong to one of following classes organized in a coarse to fine hierarchy:

World: A node containing a very coarse global map of the whole environment. The entries in the map are the nodes at one lower level: llandLoc, Locs, and Table. A movement-detector is associated with this class.

HandLoc: A special node that represents the convex closure including the hand and the held block, if any. It describes the position and the area of the closure. It. is identified by the silhouette shape and gross motion. It.s 3D position is continuously up­ dated by a hand-tracker. The state of the hand, eg holding a block or not, is also described. This infor­ mation is not acquired by vision but remembered as a result of recognizing a previous action in the cur­ rent sequence. Contains a visual search procedure.

Loc: A node that represents a convex closure including one block or an aggregation of blocks. It describes position and area. The primal visual feature rep:·~­ senting this level is "silhouette". A change-detector is associated wit.h this cla'ls.

Table : A special node representing the top of the work­ table.

Block: A precise geometrical model (B-rep) of a block . Describes shape, position, and orientation. Instan­ tiated by matching a model to the line image. An object-finder and a coplanar-detector are associated with this class.

Each class node has slots for current position, a geo­ metric model of tlw described region, and links to other nodes for other objects to which it bears relations such as supporting and holding. It also specifies associated vi­ sual feature detectors and their control procedures, and methods for interpreting visual features to generate qual­ itative events (that is, to recognize event types).

4.4 Hierarchical Parallel Automata for Motion Segmentation

Motion segmentation corresponds to transitions in hi­ erarchical parallel automata, as illustrated in Figure 4. As stated in the discussion of the main recognition pro­ cess, the three levels of automata work in parallel, but when a segmentation occurs at one level, the lower level automata are forced to terminate. This is done to main­ tain correct motion contexts. The transition events are described in Table 2. An event is detected by a corre­ sponding environment model node . It invokes an appro­ priate visual feature detector, controls the attention re­ gion and interprets the result based on the current state of the node.

4.5 The Attention "Stack"

The "attention stack" is our mechanism for implement­ ing the hierarchi,al parallel aut.omata. It. relates t.he ac­ tion model and the environment model dynamically in the context of action recognition. lt. is by manipulating the attention stack that the main recognition process controls the whole recognizer. The stack elements corre­ spond to the hierarchy of assembly motion types.

Each element of the stack can hold several "attention nodes". These nodes are generated dynamically to rep­ resent currently expected events, links to correspond­ ing action/environment model nodes, and to temporarily store detected events and feature values. The attention stack has two methods: ( 1) "Detect" queries the asso­ ciated environment. model node for the expected event and records the result, (2) "Analyze" sends the recorded event type and the feature value to the associated action model node, which provides the information needed to spe\ify attention control and the anticipated segmenta­ tion events.

5 Experimental Results

The experimental system succeeded in generating sym­ bolic descriptions of human tasks in near real-time .

Figure 5 shows a monitor display during the recogni­ tion of an table construction task. The worker's hand is tracked by multiple visual windows and an indexical feature is extracted fmm the target region. The result of recognition is displayed at. the bottom of each image. The elapsed time for the whole task was 3 min. 50 sec., excluding the computation time for initial state recogni­ tion. The worker moved his hand at low speed (typically, 10 cm/s) so that the system could follow. The system did all the processing for action segmentation and action classification on-line, and reported the recognized action type in time, before the worker completes the next. ac­ tion. Twenty-one basic level operations, were conectly segmented and dassified. Contents of the primary slots of the recognized final PLACE operation is shown in Fig­ ure 6. A summary of the whole recognized sequence is given in Figure 7.

Figure 8 shows the coplanar detector at work. In this experiment (run separately using this detector in is0la­ tion), the detector was run continuously to repeatedly check for coplanar relations. The hdd block was moved continuously and the detector reported coplanar/non­ coplanar discrimination at a. speed of I Hz .

Other assembly tasks recognized successfully by the system include; (I) pick and place with an embedded A L!GN operation (!min. 30sec.), (2) tower building (!min . 30sec., J:l operations), (3) arch construction (:.!min ., 16ops.) ( 4) building an inverted arch balanced on a center pillar (2min. 40sec., 22ops). The last task con­ tains a maintenance action, carrying a block with other blocks on it .

6 Conclusions

\Ve have demonstrated the qualitative recognition of on­ going human action sequences. Our framework includes temporal segmentation, performed by attention switch­ ing; this is initially triggered by qualitative changes in the movement of the tracked object. and is confirmed by directed visual search; and action classification, effi­ ciently done on the basis of context. ~!ependent indexical features which dirPct.ly point to qualitative action con­ cepts.

The relation between action recognition and the frame problem was shown. A hierarchical spatio-temporal at­ tention control was proposed as an incomplete but prac­ tical solution. It works both for vision processing and for the manipulation of internal representations, achiev­ ing dynamic symbol grounding to the real world.

An experimental system which recognizes simple as­ sembly tasks was presented; the system demonstrated a near real time performance . Without any cue for ac­ tion segmentation such as hit.t.ing a key or withdrawing a hand from the workspace at the end of each action, the system correctly recognized each action before the worker completed the next action.

However, our implementation relies on the simplicity of the example task and substantial improvements will have to be made for extending the system for realistic domains. First, classes of recognizable actions needs to be extended. Second, the vision modules should be im­ proved by adopting, for example, optical flow patterns for movement detection and scene segmentation, power­ ful tracking methods for 3D object movement, multiple features for objf'ct. detection and identification , active stereo vision with a zero disparity filter for object seg­ mentation, etc.

A straightforward application of the presented frame­ work will be programming robots by showing them ex­ amples of the performance of tasks, which we have demonstrated el:;;ewhere [Kuniyoshi et al., 1992]. An­ other promising direction is intelligent behavior in multi­ agent environments, where an agent recognizes other agents' actions, makes intelligent decisions and reacts to those actions, all in real time.

We expect an important modification of the presented framework: Unifying action recognition and action per­ formance. Although the presented framework assumes that the observer uses the result of action recognition for reproducing the observed task performance, the two processes (recognize and perform) are separated, using a symbolic description of action sequences as the sole in­ terface. In the modified framework, an agent observes its own actions as well as other agents' actions, under·­ standing actions in terms of what. it can do (not limited to mimicking), and sharing skills between action recog­ nition and action performance. Such agent will be able

'to learn how to recognize and to perform new sorts of actions by itself, improve recognition skills through prac­ ticing actions and vice versa.

l'vlany other problem<; are open to the researchers of qualitative action understanding, including {1) building a more general and precise theory of action recognition, (2) reasoning about the performer's intentions, {3) ex­ ploiting background knowledge such as functional de­ scriptions of physical objects.

Acknowledgements

Masayuki lnaba provided numerous important sugges­ tions and comments, especially on the implementation of the presented theory and its robotic application. Yuji Fujita, 1\azuro Hamada, l\'lakoto Fujii, Taketoshi Mori, Satoshi !washita, and Tomohiro Shibata contributed to the experimental system. Nigel ·ward provided many helpful comments on the draft of this paper.