Modeling Living Systems for Computer Vision

1 Introduction

Modeling has been a central theme of artificial intelligence. This is very evidently so in computer vision, where the mod­ eling of objects has preoccupied researchers since the dawn of the field some three decades ago [1]. A good strategy for progress on the vision problem is through physics-based modeling (see, e.g., [2]). The idea is to incorporate principles of physical dynamics into conventional geometric models in order to be able to represent not only the shapes of objects, but their physical behaviors as well. Indeed, many of the prod­ ucts of the physics-based modeling movement in computer graphics (see, e.g., [3]) are also useful in vision. This fuels our long-standing philosophy in the Visual Modeling Group at the University of Toronto that vision and graphics are mutu­ ally converse problems, and that the two fields should advance synergistically.

In this paper, I will demonstrate that we have now taken an important next step and introduced some artijicallife into the vision-graphics marriage. This enables us to set our sights on the most complex objects known-objects that are alive. I will demonstrate how recent advances in the emerging field of A-Life are inspiring fresh approaches to computer vision. 1 These advances center around the idea of artificial animals, or "animats" a term coined by Wilson [4]. In particular, I will review two of our ongoing research projects, which relate to the modeling of living systems for computer vision. The first involves the modeling of complete animals of nontrivial complexity on the evolutionary ladder, such as teleost fishes in their natural habitats (Fig. 1). The second project, involves the modeling of faces, a vitally important communicative medium of the most highly evolved living systems known-human beings (Fig. 2).

The presentation is in two parts. Sections 2 and 3 present our work on artificial fishes and animatvision. The basic idea in a nutshell is to implement, entirely in software, realistic artificial animals and to give them the ability to locomote, perceive, and in some sense understand the realistic virtual worlds in which they are situated so that they may achieve both individual and social functionality within these worlds. To this end, each animat is an autonomous agent possessing a muscle-actuated body that can locomote and a mind with motor, perception, behavior, and learning centers. The animat is endowed with functional eyes that can image the dynamic 3D virtual world onto its 2D virtual retinas. The perceptual center of the animal's brain exploits "active vision" algorithms to continually process the incoming retinal image stream in order to make sense of what it sees and, hence, to purposefully navigate its world. I hope to begin to convince the uninitiated reader that it is now within our means to implement realistic virtual worlds inhabited by artificial animals rich enough to support some serious computer vision (and AI) research.

In the second part of the paper, Sections 4 and 5, I briefly present our work on physical and anatomical modeling of human faces. Our goal has been to develop artificial faces that are capable of synthesizing realistic facial expressions. At different levels of abstraction, these hierarchical models capture knowledge about facial expression from psychology, facial anatomy and facial tissue histology, and continuum biomechanics. I will show that a generic facial model of this sort can be "personalized", or made to conform closely to individuals once the geometry and photometry of their faces has been captured by a range sensor. Finally, I will describe how this sophisticated model can be used in a model-based analysis-by-synthesis strategy to analyze facial images and image sequences, an important computer vision problem that relates to visual communication.

1 For an engaging survey of the A-Life field, see, e.g., S. Levy, Artificial Life (Pantheon, 1992). Journals such as Artificial Life and Adaptive Behavior document the state of the art.

Section 6 concludes the paper with a brief discussion of where I hope that our approach will lead us and a preview of future work.

2 Artificial Fishes

Imagine a virtual marine world inhabited by a variety of re­ alistic fishes (Fig. 1 ). In the presence of underwater currents, the fishes employ their muscles and fins to swim gracefully around immobile obstacles and among moving aquatic plants and other fishes. They autonomously explore their dynamic world in search of food. Large, hungry predator fishes stalk smaller prey fishes in the deceptively peaceful habitat. The sight of predators compels prey fishes to take evasive action. When a dangerous predator appears in the distance, similar species of prey form schools to improve their chances of sur­ vival. As the predator nears a school, the fishes scatter in terror. A chase ensues in which the predator selects victims and consumes them until satiated. Some species of fishes seem untroubled by predators. They find comfortable niches and feed on floating plankton when they get hungry. Driven by healthy libidos, they perform elaborate courtship rituals to attract mates.

Each artificial fish is an autonomous agent with a de­ formable body actuated by internal muscles. The body also harbors eyes and a brain with motor, perception, behavior, and learning centers, as Fig. 3 illustrates. Through controlled mus­ cle actions, artificial fishes are able to swim through simulated water in accordance with hydrodynamics. Their functional fins enable them to locomote, maintain balance, and maneu­ ver in the water. Thus the artificial fish model captures not just 3D form and appearance, but also the basic physics of the animal and its environment. Though rudimentary compared to real animals, the minds of artificial fishes are nonetheless able to learn some basic motor functions and carry out percep­ tually guided motor tasks. In accordance with their perceptual awareness of the virtual world, their minds arbitrate a reper­ toire of piscatorial behaviors, including collision avoidance, foraging, preying, schooling, and mating.

The details of the artificial fish model are presented in [5; 6]. I will summarize its main features in the remainder of this section.

2.1 Motor System

The motor system comprises the dynamic model of the fish including its muscle actuators and a set of motor controllers (MCs). Fig. 4(a) illustrates the biomechanical body model which produces realistic piscatorial locomotion using only 23 lumped masses and 91 elastic elements. These mechanical components are interconnected so as to maintain the structural integrity of the body as it flexes due to the action of its 12 contractile muscles.

Artificial fishes locomote like real fishes, by autonomously contracting their muscles. As the body flexes it displaces vir­ tual fluid which induces local reaction forces normal to the body. These hydrodynamic forces generate thrust that pro­ pels the fish forward. The model mechanics are governed by Lagrange equations of motion driven by the hydrodynamic forces. The system of coupled second-order ordinary differ­ ential equations are continually integrated through time by a numerical simulator. 2

The model is sufficiently rich to enable the design of motor controllers by gleaning information from the fish biomechan­ ics literature. The motor controllers coordinate muscle ac­ tions to carry out specific motor functions, such as swimming forward (swim-MC), turning left (left-turn-MC), and turning right (right-turn-MC). They translate natural control parame­ ters such as the forward speed or angle of the turn into detailed muscle actions that execute the function. The artificial fish is neutrally buoyant in the virtual water and has a pair of pectoral fins that enable it to navigate freely in its 3D world by pitching, rolling, and yawing its body. Additional motor controllers coordinate the fin actions.

2.2 Perception System

Artificial fishes are aware of their world through sensory per­ ception. The perception system relies on a set of on-board virtual sensors to gather sensory information about the dy­ namic environment. As Fig. 4(b) suggests, it is necessary to model not only the abilities but also the limitations of animal perception systems in order to achieve natural sensorimotor behaviors. The perception center of the brain includes a per­ ceptual attention mechanism (see [7] for a review of attention mechanisms) which allows the artificial fish to sense the world in a task-specific way, hence filtering out sensory information superfluous to its current behavioral needs. For example, the artificial fish attends to sensory information about nearby food sources when foraging. Our early artificial fishes-those that are not equipped with the active vision system described in Section 3-employ simulated perception, a "perceptual ora­ cle" which satisfies the fish's sensory needs by directly in­ terrogating the 3D world model-the fish can directly access the geometric and photometric information that is available to the graphics rendering engine, as well as object identity and dynamic state information about the physics-based world model.

2 The artificial fish model achieves a good compromise between realism and computational efficiency. For example, the implemen­ tation can simulate a scenario with 10 fishes, 15 food particles, and 5 static obstacles at about 4 frames/sec (with wireframe rendering) on a Silicon Graphics R4400 lndigo 2 Extreme workstation. More complex scenarios with large schools of fish, dynamic plants, and full color texture mapped GL rendering at video resolution can take 5 seconds or more per frame.

2.3 Behavior

The behavior center of the artificial fish's brain mediates be­ tween its perception system and its motor system. An in­ tention generator, the fish's cognitive faculty, harnesses the dynamics of the perception-action cycle. The innate charac­ ter of the fish is established by a set of habits that determine if it is male or female, predator or prey, etc. At each simulation time step, the intention generator takes into account the habits of the fish and the incoming stream of sensory information to generate dynamic goals for the fish, such as to avoid an ob­ stacle, to hunt and feed on prey, or to court a potential mate. It ensures that goals have some persistence by exploiting a single-item memory. Persistence makes sustained behaviors such as foraging, schooling, and mating more robust. The in­ tention generator also controls the perceptual attention mech­ anism. At every simulation time step, the intention generator activates behavior routines that attend to sensory informa-. tion and compute the appropriate motor control parameters to carry the fish one step closer to fulfilling its current intention. The behavioral repertoire of the artificial fish includes primi­ tive, reflexive behavior routines, such as obstacle avoidance, as well as more sophisticated motivational behavior routines such as schooling and mating whose activation depends on the dynamic mental state of the fish as represented by hunger, fear, and libido mental variables (see [5] for the details).

2.4 Learning

The learning center of its mind enables the artificial fish to learn how to locomote through practice and sensory reinforce­ ment. Through optimization, the motor learning algorithms discover muscle controllers that produce efficient locomotion. Muscle contractions that produce forward movements are "re­ membered". These half-success then form the basis for the fish's subsequent improvement in its swimming technique. Their brain's learning center also enable these artificial ani­ mals to train themselves to accomplish higher level sensorimo­ tor tasks, such as maneuvering to reach a visible target (see [5; 8] for the details).

2.5 Modeling Fonn and Appearance

Of course, we want our animats to capture the form and ap­ pearance of real fishes with considerable visual fidelity. Visual fidelity is especially important in the application of our an­ imats to computer vision, which I will describe shortly. To this end, photographs of real fishes, such as the one shown in Fig. 5(a), are converted into 3D spline (NURBS) surface body models (Fig. 5(b)). The digitized photographs are ana­ lyzed semi-automatically using a "snake-grid" tool which is demonstrated in Fig. 5(d-e) on a different fish image. The grid of snakes [9] floats freely over an image. The border snakes adhere to intensity edges demarcating the fish from the background, and the remaining snakes relax elastically to cover the imaged fish body. This yields a smooth, nonuniform coordinate system (Fig. 5(e)) for mapping the texture onto the spline surface to produce the final texture mapped fish body model (Fig. 5(c)).

3 Animat Vision

The psychologist J.J. Gibson stressed in pre-computational terms the importance of modeling the active observer situated in the dynamic environment [10]. Versions of this paradigm suitable for mainstream computer vision were introduced in the seminal papers of Bajcsy [ 11] and Ballard [ 12] under the names of active perception and animate vision, respectively. 3

3 "Animat" vision should not be confused with Ballard's "ani­ mate" vision; the latter does not involve artificial animals.

The active vision approach has developed into a prevailing paradigm [13; 14; 15; 16].

3.1 Problems with the "Hardware Vision" Mindset

As active vision is practiced in most labs today, however, it is in reality little more than technology-driven "hardware vision". To be sure, applications-minded researchers have le­ gitimate reasons for building robot vision systems, but the nec­ essary hardware paraphernalia-CCD cameras, pan-tilt units, ocular heads, frame-rate image processors, mobile platforms, manipulators, controllers, interfaces, etc.-can be expensive to fabricate or acquire commercially and a burden to maintain in good working order.

The animat vision methodology that we propose in [ 17] can potentially liberate a significant segment of the computer vision research community from the tyranny of robot hard­ ware. It addresses the needs of scientists who are motivated to understand and ultimately reverse engineer the powerful vi­ sion systems found in higher animals. These researchers are well aware that animals do not have CCD chip eyes, electric motor muscles, and wheel legs. That is to say, they realize that readily available hardware systems are terrible models of biological animals. For lack of a better alternative, however, they have been struggling with inappropriate hardware in their ambition to understand the complex sensorimotor functions of real animals. Moreover, their mobile robots typically lack the compute power necessary to achieve real-time response within a fully dynamic world while permitting active vision research of much significance.

Artificial animals such as the fish are active "vehicles" in the sense of Braitenberg [18]. We believe that they are as appropriate for grounding active vision systems as are the hardware "mobots" that have come out of the situated robotics work of Brooks and his group [19; 20] and have been an inspiration to numerous other robotics groups (see, e.g., the compilation [21]). Undeniably, however, efforts to equip real­ time mobile robots with general-purpose active vision systems have been hampered by the hardware and the relatively modest abilities of on-board processors. ·

3.2 Get A-Life!

I will now describe a zoomimetic approach to vision [17] which is made possible by the confluence of

1. advanced physics-based artificial life modeling of ani­ mals, as I presented it in the previous section
2. photorealistic computer graphics rendering, especially as implemented in modern 3D graphics workstations, and
3. active computer vision algorithms.

Our animat vision approach offers an alternative strategy for developing biologically inspired active vision systems that circumvents the aforementioned problems of hardware vision. The animat vision concept is realized with realistic artificial animals and active vision algorithms implemented entirely in software on 3D graphics workstations. Animat vision offers several additional advantages:

• One can slow down the "cosmic clock" of the virtual world relative to the cycle time of the CPU on which it is being simulated. This increases the amount of compu­ tation that each agent can consume between clock ticks without unduly retarding the agent's responses relative to

the temporal evolution of its virtual world. This in turn permits the development and evaluation of new and/or computationally complex active vision algorithms that are not presently implementable in real-time hardware.

• The quantitative photometric, geometric, and dynamic information that is needed to render the virtual world is available explicitly. Generally, tbe animats are privy to none of this environmental ground truth data, but must glean visual information the hard way-from their retinal image streams. However, the readily available ground truth can be extremely useful in assaying the effective accuracy of the vision algorithms or modules under de­ velopment.

Our challenge has been to synthesize a prototype active vi­ sion system for the fish animat which is based solely on retinal image analysis [ 17]. The vision system should be extensible so that it will eventually support the broad repertoire of indi­ vidual and group behaviors of artificial fishes. It is important to realize that we need not restrict ourselves to modeling the perceptual mechanisms of real fishes. In fact, the animat vision paradigm applies to animats that model any animal­ even a human being-to the level of fidelity that the artificial fish models a real fish. Indeed, the animat vision system that we have developed does not attempt to model fish perception [221. Given a piscine animat that is an active observer of its world, we have found it interesting and challenging to endow the animat with human-like retinal imaging capabilities!

3.3 Active Vision System

The basic functionality of the animat vision system starts with binocular perspective projection of the color 3D world onto the animat's 2D retinas. Retinal imaging is accomplished by photorealistic graphics rendering of the world from the animat's point of view. This projection respects occlusion relationships among objects. It forms spatially variant visual fields with high resolution foveas and low resolution periph­ eries. Based on an analysis of the incoming color retinal image stream, the visual center of the animat's brain supplies saccade control signals to its eyes and stabilize the visual fields during locomotion, to attend to interesting targets based on color, and to keep a moving/deforming target fixated. The artificial fish is thus able to approach and track other artificial fishes using visual feedback. Eventually its arsenal of active vision algorithms will enable it to forage, evade predators, find mates, etc.

Fig. 6 is a block diagram of the active vision system showing two main modules that control foveation of the eyes and retinal image stabilization.

Eyes and Retinal Imaging

The artificial fish has binocular vision. The movements of each eye are controlled through two gaze angles ( fJ, <P) which specify the horizontal and vertical rotation of the eyeball, respectively. The angles are given with respect to the head coordinate frame, such that the eye is looking straight ahead when fJ = <P = 0° .

Each eye is implemented as four coaxial virtual cameras to approximate . the spatially nonuniform, foveal/peripheral imaging capabilities typical of biological eyes. Fig. 7(a) shows an example of the 64 x 64 images that are rendered by the four coaxial cameras (using the GL library and SGI

Gaze Control For One Eye

Figure 6: The animat vision system. The flow of the algorithm is from right to left. A: Update gaze angles ( fJ, <P) and saccade using these angles, B: Search current level for model target and if found localize it, else search lower level, C: Select level to be processed (see text), F: Reduce field of view for next level and render, M: Compute a general translational displacement vector (u, v) between images I(t- 1) and I(t), S: Scale the color histogram of the model for use by the current level.

graphics pipeline) of the left and right eye. The level/ = 0 camera has the widest field of view (about 120°) and the low­ est resolution. The resolution increases and the field of view decreases with increasing/. The highest resolution image at level/ = 3 is the fovea and the other images form the visual periphery. Fig. 7(b) shows the 512 x 512 binocular retinal images composited from the coaxial images at the top of the figure. To reveal the retinal image structure in the figure, we have placed a white border around each magnified component image.

The advantages of the multiresolution retina are significant. Vision algorithms which process the four 64 x 64 component images are 16 times more efficient than those that process a uniform 512 x 512 retinal image.

Foveation by Color Object Detection

The mind of the fish stores a set of color models of objects that are of interest to it. For instance, if the fish is by habit a predator, it would possess models of prey fish. The models are stored as a list of 64 x 64 RGB color images in the fish's memory.

To detect and localize any target that may be imaged in the low resolution periphery of its retinas, the animat vi­ sion system of the fish employs an improved version of a color indexing algorithm proposed by Swain [23]. Since each model object has a unique color histogram signature, it can be detected in the retinal image by histogram intersection and localized by histogram backprojection.

Saccadic Eye Movements

When a target is detected in the visual periphery, the eyes will saccade to the angular offset of the object to bring it within the fovea. With the object in the high resolution fovea, a more accurate foveation is obtained by a second pass of histogram backprojection. A second saccade typically centers the ob­ ject accurately in both left and right foveas, thus achieving vergence.

Module A in Fig. 6 performs the saccades by incrementing the gaze angles ( 8, tjJ) in order to rotate the eyes to the required gaze direction.

Visual Field Stabilization using Optical Flow

It is necessary to stabilize the visual field of the artificial fish because its body undulates as it swims. Once a target is verged in both foveas, the stabilization process (Fig. 6) assumes the task of keeping the target foveated as the fish locomotes.

Stabilization is achieved by computing the overall transla­ tional displacement ( u, v) oflight patterns between the current foveal image and that from the previous time instant, and up­ dating the gaze angles to compensate. The displacement is computed as a translational offset in the retinotopic coordi­ nate system by a least squares minimization of the optical flow between image frames at times t and t - 1 [24].

The optical flow stabilization method is robust only for small displacements between frames. Consequently, when the displacement of the target between frames is large enough that the method is likely to produce bad estimates, the foveation module is invoked to re-detect and re-foveate the target as described earlier.

Each eye is controlled independently during foveation and stabilization of a target. Hence, the two retinal images must be correlated to keep them verged accurately on the target. Referring to Fig. 8, the vergence angle is Bv = ( 8 R - 8 L) and its magnitude increases as the fish comes closer to the target. Therefore, once the eyes are verged on a target, it is straightforward for the fish vision system to estimate the range to the target from the gaze angles.

3.4 Vision-Guided Navigation

The fish can use the gaze direction for the purposes of navi­ gation in its world. In particular, it is natural to use the gaze angles as the eyes are fixated on a target to navigate towards the target. The 8 angles are used to compute the left/right tum angleOp shown in Fig. 8, and the tjJ angles are similarly used to compute an up/down tum angle tjJ p . The fish's tum motor con­ trollers are invoked to execute a left/right tum-left-tum-Me for an above-threshold positive 8 p and right -tum-MC for neg­ ative Op (see Section 2)-with lOp I as parameter. Up/down tum motor commands are issued to the fish's pectoral fins, with an above-threshold positive tPP interpreted as "up" and negative as "down".

The problem of pursuing a moving target that has been fixatyd in the foveas of the fish's eyes is simplified by the gaze control mechanism described above. The fish can ro­ bustly track a target in its fovea and follow it around by using the tum angles ( 8 p, tjJ p) computed from the gaze angles that are continuously updated by the foveation/stabilization algorithms.

We have carried out numerous experiments in which the moving target is a reddish prey fish whose color histogram model is stored in the memory of a predator fish equipped with the active vision system. Fig. 9 shows plots of the gaze angles and the turn angles obtained over the course of 100 frames in a typical experiment as the predator is fixated on and actively pursuing a prey target. Fig. 10 shows a sequence of image frames acquired by the fish during its navigation (monochrome versions of only the left retinal images are shown). Frame 0 shows the target visible in the low res­ olution periphery of the fish's eyes (middle right). Frame 1 shows the view after the target has been detected and the eyes have performed a saccade to foveate the target (the scale ·difference of the target after foveation is due to perspective distortion). The subsequent frames show the target remaining fixated in the fovea despite the side-to-side motion of the fish's body as it swims towards the target.

The saccade signals that keep the predator's eyes fixated on its prey as both are swimming are reflected by the undulatory responses of the gaze angles in Fig. 9. The figure also shows that the vergence angle increases as the predator approaches its target (near frame 100). In comparison to the() angles, the <P angles show little variation, because the fish does not undulate vertically very much as it swims forward. It is apparent from the graphs that the gaze directions of the two eyes are nicely correlated.

Note that in frames 87-117 of Fig. 10, a yellow fish whose size is similar to the target fish passes behind the target. In this experiment the predator was instructed to be totally un­ interested in and not bother to foveate non-reddish objects. Because of the color difference, the yellow object does not distract the fish's gaze from its reddish target. This demon­ strates the robustness of the color-based fixation algorithm.

4 Artificial Faces

I will now shift gears and discuss the modeling of human animals, focusing on the important and challenging problem of modeling human faces.

The human face has attracted much attention in several disciplines, including psychology, computer vision, and com­ puter graphics. Psychophysical investigations clearly indicate that faces are very special visual stimulii. Psychologists have studied various aspects of human face perception and recogni­ tion [25; 26]. They have also examined facial expression-the result of a confluence of voluntary muscle articulations which deform the neutral face into an expressive face. The facial pose space is immense. The face is capable of generating on the order of 55,000 distinguishable facial expressions with about 30 semantic distinctions. Studies have identified six primary expressions that communicate anger, disgust, fear, happiness, sadness, and surprise in all cultures.

Ekman and Friesen's "Facial Action Coding System" (FACS) provides a quantification of facial expressions [27]. The FACS quantifies facial expressions in terms of 44 "action units" (AU) involving one or more muscles and associated activation levels.

4.1 A Functional Facial Model

We have developed a hierarchical model of the face which provides natural control parameters and is efficient enough to run at interactive rates [28]. Conceptually, the model decom­ poses into six levels of abstraction. These representational levels encode specialized knowledge about the psychology of human facial expressions, the anatomy of facial muscle struc­ tures, the histology and biomechanics of facial tissues, and facial skeleton geometry and kinematics:

1. Expression. At the highest level of abstraction, the face model executes expression (or phoneme) commands. For instance, it can synthesize any of the six primary expres­ sions within a specific time interval and with a specified degree of emphasis.
2. Control. A muscle control process, a subset of Ekman and Friesen's FACS, translates expression (or phoneme) instructions into a coordinated activation of actuator groups in the facial model.
3. Muscles. As in real faces, muscles comprise the basic actuation mechanism of the model. Each muscle model consists of a bundle of muscle fibers. When fibers con­ tract, they displace their points of attachment in the facial tissue or the jaw.
4. Physics. The face model incorporates a physical approx­ imation to human facial tissue. The tissue model is a lattice of point masses connected by nonlinear elastic springs. Large-scale synthetic tissue deformations, sub­ ject to volume constraints, are simulated numerically by continuously propagating through the tissue lattice the stresses induced by activated muscle fibers.
5. Geometry. The geometric representation of the facial model is a non-uniform mesh of polyhedral elements whose sizes depend on the curvature of the neutral face. Muscle-induced synthetic tissue deformations distort the neutral geometry into an expressive geometry.
6. Images. After each simulation time step, standard visu­ alization algorithms implemented in dedicated graphics hardware render the deformed facial geometry in accor­ dance with viewpoint, light source, and skin reflectance information to produce the lowest level representation in the modeling hierarchy, a continuous stream of facial images.

The hierarchical structure of the model encapsulates most of the complexities of the underlying representations, relegat­ ing the details of their computation to automatic procedures. At the higher levels of abstraction, our face model offers a semantically rich set of control parameters which reflect the natural constraints of real faces.

Our synthetic facial tissue model is motivated by histology and tissue biomechanics. Human skin has a nonhomogeneous and nonisotropic layered structure consisting of the epidermis, dermis, subcutaneous fatty tissue, fascia, and muscle layers. The synthetic tissue is a deformable lattice, an assembly of discrete finite elements (see [28] for the details).

4.2 Personalizing the Functional Model

We have developed a highly automated approach to construct­ ing realistic, functional models of human heads [29]. These physics-based models are anatomically accurate and may be made to conform closely to specific individuals. Currently, we begin by scanning a subject with a laser sensor which circles the head to acquire detailed range and reflectance in­ formation. Next, an automatic conformation algorithm adapts a triangulated face mesh of predetermined topological struc­ ture to these data. The generic mesh, which is reusable with different individuals, reduces the range data to an efficient, polygonal approximation of the facial geometry and supports a high-resolution texture mapping of the skin reflectivity.

The conformed polygonal mesh forms the epidermal layer of a physics-based model of facial tissue. An automatic algo­ rithm constructs the multilayer synthetic skin and estimates an underlying skull substructure with a jointed jaw. Finally, the algorithm inserts synthetic muscles into the deepest layer of the facial tissue. These contractile actuators, which emulate the primary muscles of facial expression, generate forces that deform the synthetic tissue into meaningful expressions. To increase realism, we include constraints to emulate tissue in­ compressibility and to enable the tissue to slide over the skull as real skin does.

Fig. 11 illustrates the aforementioned steps. The figure shows a 360° head-to-shoulder scan of a woman, "Heidi," acquired by a Cyberware Color 3D Digitizer. The data set consists of a radial range map (Fig. 11(a)) and a registered RGB photometric map (Fig. 1l(b)). The range and RGB maps are high-resolution 512 x 256 arrays in cylindrical coordinates, where the x axis is the latitudinal angle around the head and the y axis is vertical distance. Fig. ll(c) shows the generic mesh projected into the 2D cylindrical domain and overlayed on the RGB map. The triangle edges in the mesh are elastic springs, and the mesh has been conformed automatically to the woman's face using both the range and RGB maps [29]. The nodes of the conformed mesh serve as sample points in the range map. Their cylindrical coordinates and the sampled range values are employed to compute 3D Euclidean space coordinates for the polygon vertices. In addition, the nodal coordinates serve as polygon vertex texture map coordinates into the RGB map. Fig. 11 (d) shows the 3D facial mesh with the texture mapped photometric data.

Once we have reduced the scanned data to the 3D epider­ mal mesh of Fig. 11(d), we can assemble a physics-based face model of Heidi, including the synthetic skin (Fig. 11 (e) shows a skin patch undergoing deformation) and muscles (Fig. 11(e) shows the contractile muscles (vectors) underneath the epi­ dermal mesh). Fig. 11(g-j) demonstrates the resulting facial model producing animated expressions by contracting facial muscles. The same technique was applied to animate the facial model of George shown in Fig. 2.

5 Model-Based Facial Image Analysis

Facial image analysis and synthesis is necessary for numerous applications. Among them is low bandwidth teleconferencing which may involve the real-time extraction of facial control parameters from live video at the transmission site and the reconstruction of a dynamic facsimile of the subject's face at a remote receiver. Teleconferencing and other applications require facial models that are computationally efficient and also realistic enough to accurately synthesize the various nu­ ances of facial structure and motion. We have argued that the anatomy and physics of the human face, especially the ar­ rangement and actions of the primary facial muscles, provide a good basis for facial image analysis and synthesis [30].

The physics-based anatomically motivated facial model has allowed us to develop a new approach to the analysis of dynamic facial images for the purposes of estimating and resynthesizing dynamic facial expressions [30]. Part of the difficulty of facial image analysis is that the face is highly de­ formable, particularly around the forehead, eyes, and mouth, and these deformations convey a great deal of meaningful in­ formation. Techniques for tracking the deformation of facial features include "snakes" [9]. Motivated by the anatomically consistent musculature in our model, we have considered the estimation of dynamic facial muscle contractions from video sequences of expressive faces. We have developed an anal­ ysis technique that uses snakes to track the nonrigid motions of facial features in video. Features of interest include the eyebrows, nasal furrows, mouth, and jaw in the image plane. We are able to estimate dynamic facial muscle contractions directly from the snake state variables. These estimates make appropriate control parameters for resynthesizing facial ex­ pressions through a generic face model at real-time rates.

Fig. 13 shows a plot of the estimated muscle contractions versus the frame number. They are input to the physics-based model as a time sequence. The model resynthesizes the facial expression. Three rendered images are shown in Fig. 12(c).

6 Where do we go from here?

I have presented two of our ongoing research projects that span the fields of computer vision, artificial life, and computer graphics. The projects are related in that they involve the de­ velopment of nontrivial models ofliving systems. The models are founded upon computational physics. I have demonstrated applications of each of the models to computer vision.

To summarize, on one front, we have made significant progress over the past two years in developing a model which captures the essential features of most living systems­ biomechanics, locomotion, perception, behavior, and learn­ ing. We are now using this piscatorial model as a situated virtual robot for active vision research. It is my hope that the active vision systems that we are synthesizing in this way will be relevant in whole or in part to physical robotics. It seems to me that virtual animats in their dynamic world can serve as an useful proving ground for theories that profess sensorimotor competence in animal or robotic situated agents. 4

On another front, we have been able to generate functional models of people's heads and use them for model-based facial image analysis. At the biomechanical and anatomical levels, the face models are as faithful to human faces as the fish models are to real fishes. Unlike artificial fishes, however, the disembodied artificial heads do not yet have a much of a brain-just a simple motor center that blinks eyelids, moves the eyes, flexes the neck, and coordinates the facial muscles to produce meaningful expressions.

We would like to construct a brain model for the artificial heads that is as at least as comprehensive as the brain of the artificial fish, in the sense that it should be capable of dynamic perception and cognitively motivated behavior depending on environmental influences. We can get to this goal in an inter­ esting way.

An Artificial Mermaid: A logical next step, given what we have implemented already, would be to couple the artificial head model to the posterior of the artificial fish model with an anthropomorphic torso to create an artificial mermaid. The mermaid will be able to locomote through its virtual under­ water world as the fishes now do. Using an animat vision system, it will be able to perceive and interact with fish and other mermaids. Unlike fish, however, the mermaid will have some of the expressive and behavioral capabilities of a human. Far from being frivolous, this virtual creature could serve to smoothly bridge the gap between the humble cognitive abili­ ties of an artificial fish and-maybe some day-human level intelligence.

The Artificial Life of a Virtual Human: Naturally, an exciting long-term goal that should elicit little controversy within the AI community is to develop an intelligent artificial human that is at least as convincing on virtual terra firma as our artificial fishes are in their virtual seas. It seems to me that we are well on the way to this end. To achieve this goal, how­ ever, more progress will obviously be necessary on several challenging problems, not excluding the AI problem.

To be continued .. .

4 Doom vision: As a further test of the animat vision paradigm, we are developing an active vision system, similar to the one in artificial fishes, within an autonomous agent situated in a "doom" world ("Doom" is an amazingly popular video game). The challenge is for this agent to assume the role of the human doom player, given the same dynamic graphical image(s) that a human player would see displayed on the screen. The agent's "brain" will interpret the incoming retinal image stream and generate motor commands, analogous to the keyboard commands a human player would issue, to locomote through the amusingly hostile doom world. A successful doom agent would be able to do what a skilled human player does­ explore, accumulate points, and avoid being killed.

Acknowledgements

I would like to thank my students for their important contribu­ tions to the research described herein. Xiaoyuan Tu developed the artificial fish animat. Tamer Rabie developed the animat vision system. Radek Grzeszczuk developed the animat learn­ ing algorithms. Yuenchen Lee developed the physics-based face model. I thank Keith Waters for our years of collaboration on facial modeling. I also thank the many persons who have discussed and debated with me some of the ideas contained herein, especially John Tsotsos (who also provided valuable comments on a draft), Geoffrey Hinton, and Allan Jepson. Range/RGB facial data were provided courtesy ofCyberware, Inc., Monterey, CA. The research described herein was sup­ ported by grants from the Natural Sciences and Engineering Research Council of Canada and by the ARK (Autonomous Robot for a Known environment) Project, which receives its funding from PRECARN Associates Inc., Industry Canada, the National Research Council of Canada, Technology On­ tario, Ontario Hydro Technologies, and Atomic Energy of Canada Limited. The author is a fellow of the Canadian In­ stitute for Advanced Research.