Rule-Structured Facial Animation System

1 Introduction

Facial animation is part of 3D character animation con­ cerned with the modeling, motion control and rendering of human faces. As the main communication channel, human face is a very sensitive object which is moreover never still. Facial and audio channels complement and support each other: visual channel can clarify and make unambiguous what is being said. While talking, people not only use their lips but their eyebrows may raise and create expressive wrinkles which will amplify the con­ tent of the speech; they may turn the head to underline a specific word; their eyes may move or blink, etc. It is the accumulation of such synchronized details which give expression and life to synthetic animation.

Earlier systems [Parke, 1989] used a set of confor­ mation and expression parameters to perform the ani­ mation. Simulation of muscles of the face and of the propagation of their movement [Platt, 1985], [Waters, 1987] produce more accurate facial actions. Modelling

·ciS department, Philadelphia, PA 19104 U.S.A

1 Domaine de Voluceau, BP 105, 78153 Le Chesnay, France

IDomaine de Voluceau, BP 105, 78153 Le Chesnay, France

Marie-Luce Viaudt INRI A 1 uce@ bora.inria.fr Hussein Yahia~ INRI A hussein @bora.inria.fr

the external and interface structures of the skin leads to a computer simulation of some plastic surgery operations [Pieper, 1992]. Multi-layer approach [Kalra et al., 1991] or addition of some speech parameters [Hill et al., 1988], [Lewis and Parke, 1987], lNahas, 1988] were included in animation systems to produce automatic lip synchro­ nization. Their approach was to express the animation by a set of rules where every element (of the sound and of the face) can be modified interactively through the use of parameters. To gain higher-level control over facial animation, techniques of extracting contours over desig­ nated areas on a real actor's face recorded on video were developed [Terzopoulos and Waters, 1991].

Facial expressions are often very subtle. The observa­ tion of details such as expressive wrinkles, and conver­ sational signals in the speech, is an important feature of understanding human face . Thus, in order to improve facial animation systems, perceiving this facial language and its interaction with intonation is one of the most important steps. The consideration and synchronization of visual and audio channels enhance animation system.

We present in this paper a system in which secondary effects of the movement are modelled according to the primary deformation of the muscles and their effects on the skin. We first characterize the various channels of facial expressions and the intonational notation we are using. We explain the assumption of our system and its methodology. We then present our facial model. We define in our context wrinkles and bulges. We describe how we have included then in our animation system.

1.1 Facial Notation

We are using the FACS notation (Facial Action Coding System) created by P. Ekman and W. Friesen [Ekman and Friesen, 1978]. It is based on anatomical studies. Since facial movement is due to muscle action, a system was derived to describe visible facial expressions (emo­ tional or conversational) at the muscle level. An action unit AU, basic element of this system, describes the ac­ tion produced by one or a group of related muscles.

This notation is relevant in our case since it allows us to work at the AUs level and not at the regions or muscles level; it implies then the independence of the facial model we are using.

2 Analysis of Facial Action and Vocal Parameters

Our main goal is to look at the coordinated motion during conversation. Our method is based on finding the link between the spoken intonation, the transmitted information in the given context, and the facial move­ ments.

To counterbalance the difficulties involved in manip­ ulating manually the action of each muscle, our system offers to the user a higher level of animation by lip syn­ chronization and by an automatic computation of the facial expressions related to the patterns of the voice. Not all facial expressions are used to characterize emo­ tions. Some can highlight a word, underline a pause; some are tied to the intonation of the voice. Our first step was to elaborate a repertory of such facial actions called channels.

2.1 Facial Channel

Facial actions can be clusterized depending on their func­ tionality [Ekman, 1979].

phonemic channel : lip shapes and coarticulation.

intonational channel :

CONVERSATIONAL SIGNALS: when occur­ ring on accented items or on emphatic segments, they clarify and support what is being said. A stressed element is often accompanied not by a par­ ticular movement but by an accumulation of rapid movements (such as more pronounced mouth posi­ tion, blinks or rapid head movements).

PUNCTUATORS: they can reduce the ambi­ guity of speech, grouping or separating sequence of words into discrete unit phrases. A boundary point (such as a comma) will be underlined by slow move­ ment and a final pause will coincide with stillness [Hadar et a/., 1983]. Eyeblinks can occur also dur­ ing pauses [Argyle and Cook, 1976].

REGULATORS: they help the interaction be­ tween speaker-listener, they control the flow of speech. Head and eye movements are coordinated to synchronize speech and organize communication. Speaker-Turn system groups them into four sets [Duncan, 1974].

MANIPULATORS: they correspond to the bi­ ological needs of the face (like blinking to keep the eyes wet) .

emotional channel : Emotions are mainly displayed on the face and through the voice. Six emotions (anger, disgust, fear, happiness, sadness and sur­ prise) were found to have universal facial expres­ sions [Ekman, 1979] corresponding to prototypes. We have chosen to study all of them [Pelachaud et a/., 1991a].

We have excluded all movements not related directly to the pattern of the voice such as intentional, deliberate actions used to communicate.

2.1.1 Separability of the Channels

We differentiate facial actions by their functions (con­ versational signals versus punctuators) and not by the facial features it involved in the actions (eyebrows ver­ sus nose wrinkling). Therefore channels are functionally independent with each other even though they might in­ volve the same facial regions (i.e. an eyebrow movement can accompany an accented item or be part of an emo­ tive signal). This independence allows us to express any facial action of a channel as a set of AUs which has the consequence to make the computation independent of the facial model.

2.2 Vocal Parameters

A parallel between the syntax of sentences and supraseg­ mental features is defined. Suprasegmental features are added by the speaker to the text and their meanings are directly related to the strength of the conveyed af­ fect as a linear function of the degree of arousal. The listener detects the speaker's emotion from prosodic fea­ tures. Those are the pitch (while frequency is a physical property of a sound, pitch is a subjective one), loudness (the perceived intensity of a sound), pitch contour (the global envelope of the pitch), tempo (rate of speech) and pause.

2.2.1 Specification of the Intonation

To define the syntactic structure of intonation, we are using an extension of Janet Pierrehumbert 's nota­ tion [Pierrehumbert and Hischberg, 1987]. Under this definition, intonation consists of a linear sequence of accents made from two tones (H and L for high and low tones respectively) . Utterances are decomposed into intonational and intermediate phrases. Different into­ national "tunes" composed of these elements are used to convey various discourse-related distinctions of ''fo­ cus": old/new information, contrast and propositional attitude.

We can represent the decomposition of an utterance into intonational (or intermediate) phrases by brackets. The choice of intonational bracketing is obtained by the context the sentence is uttered and on the meaning of the utterance (what the speaker wants to focus on, what sjhe considers as new information versus old) [Steedman, 1991].

The input utterance "Julia prefers popcorn" has the following bracketing and set of accents in the context defined by the question:

Question: Well, what about JUlia?

What does SHE prefer?

Answer: (JUlia prefers) (POPcorn).

Accent: (L+H·

LH%) (H· LL%).

(Nota Bene: * and %represent respectively pitch and boundary accents) .

3 Relation Voice/Facial Movement

We present here the principles and parametrized schemes we are using in our system.

3.1 Synchrony

An important property linking intonation and facial ex­ pression is the existence of synchrony between them. That is changes occurring in speech and facial move­ ments should appear at the same time. Synchrony oc­ curs at all levels of speech (a blink is more adapted to phoneme level, while eyebrows action to the word level) .

This is the basic principle which regulates the computa­ tion of our facial animation.

3.2 Modularity

The computation over the different channels is per­ formed by a set of rules. These rules are derived from physiological, psycholo~ical, and linguistic considera­ tions [Pelachaud, 1991bj. This scheme allows the user to modify, add, or delete a rule for one channel with­ out modifying the other part of the system. Considering different channels for facial movement offers the possi­ bility for researchers to experiment with their individual significance. Switching off one of the component chan­ nel is a way to analyze the meaning and information it conveys. For the same reasons, the system will allow modular refinement of each channel.

3.2.1 Parameters of Facial Action

Facial expressions follow the flow of speech but they are also tied to the speaker's individuality. Speakers dif­ fer in their way to punctuate speech and in the number of displayed action. Therefore two independent variables define a facial action, its type (set of AUs) and its time of occurrence (in the spoken utterance).

In order to vary the manifestation of speaker's indi­ viduality (submissive, persuasive, or ... ) and attitude (what s/he wants to convey, such as, for instance, irony or politeness) the user can specify particular set of type and timing data for the facial actions.

4 Rule-Based Program

In these sections we describe the system of rules we are using to compute the facial expressions involved in the animation.

4.1 Different Levels of Rules

Following our principle of synchrony, each rule involves a particular level. All the rules defining the lip shapes are at the phoneme level. Blinks (appearing as conver­ sational signals or punctuators) involve the same level. The various rules computing the existence and timing of blinks determine on which phonemes the eyes start to close, on how many phonemes the eyes remain closed, and finally on which phoneme the eyes start to open again.

On the other hand, rules related to conversational sig­ nal work at the word level (or syllable level depending on the speech-rate). They compute the beginning and end of an action not by scanning the utterance phoneme by phoneme but word by word (or syllable by syllable).

Regulators involving mainly head shifts act on parts of the utterance; for group of related utterances, head movements show identical pattern.

The rules defining the types of facial action to express an emotion is at the utterance level; i.e. it alters the entire animation.

Emotion does not modify the shape of the contour of an utterance, i.e., it does not affect either the type or the placement of the accents (which are defined by the context of an utterance, what is new /old information to the speaker). This property allows us to compute ev­ ery facial action corresponding to the given intonational pattern.

4.2 Rules of Occurrence of Action

The rules corresponding to the emotion are evaluated on the top of the already found rules. They determine the effective existence of a facial movement (a sad person shows less movement than a frightened person) . Some types of facial action are emotion dependent and are therefore computed at this level. For instance, a dis­ gusted person has tendency to punctuate silences, with nose wrinkling, while a happy person smiles.

Choosing interactively an emotion not belonging to the basic set (of the six defined emotions) is not avail­ able at the present state of this study yet. But it can be obtained by adding to the main set of rules, the rules defining the occurrence qf facial actions and the list of AUs for the chosen emotion; that is to specify for ex­ ample if blinks should occur on pauses and accents.

4.3 Coarticulation Rules

There is not a unique equivalence between a phoneme and a particular lip shape due to the phenomenon of coarticulation. Coarticulation occurs due to the over­ lap of phonemic segments during their production. The boundary between segments are blurred. No complete set of rules exists. We have implemented look-ahead model [Kent and Minifie, 1977] which considers articula­ tory adjustment on a sequence of consonants followed or preceded by a vowel. To counterbalance some of the un­ solved problems due to the incompleteness of this model , geometric and temporal constraints were integrated in the set of rules. They are applied on every phonemic element belonging to context-dependent clusters [Jef­ fers and Barley, 1971]. Some rules were defined to look at how consecutive actions are performed; they modify the intensity of an action accordingly (geometric con­ straints). Relaxation and contraction time of a muscle (temporal constraint) consider if the current speech pos­ ture has time to relax before the next speech posture (or, respectively, to contract after the previous one) oth­ erwise the next speech posture is influenced by the re­ laxation of the current one (some action of the current speech posture remains over the next one) [Pelachaud, 1991b].

4.4 Attenuation Rules

The final animation is obtained by adding the list of facial action (set of AUS) occurring for each channel (AUs as defined in FACS work in an additive man­ ner) . Channels might involve the same facial regions. In such a case, attenuation rules are applied (it is done mainly for lips, head and eyes gestures) . These rules at­ tenuate conflicting movement rather than eliminate one action. For instance, the head may turn slowly during a sequence of words, while nodding (changing ups-and­ downs direction at every phoneme). The intensity of these movements are modified (they decrease) but not their existence (both movements are still occurring).

5 Execution Step

The input of our program is explained here as well as the various steps of computation to produce the final set of facial expressions for the animation.

5.1 Assumption for the Input

The input of our program is an utterance with its in­ tonational pattern specified. For the moment, we are using recorded natural speech to guide our animation. After recording a sentence, we extract from its spectro­ gram the timing of each phoneme and pause. We plan to use analysis-and-resynthesis methods to automate the determination of paralanguage parameters and phoneme timing.

At the beginning of the file , the user should specify the emotion. The intensity of emotion (a number between 0, for minimum intensity, and 1, for maximum intensity) should also be indicated. Furthermore, we assume that the input contains the decomposition of the utterance into its bracketed elements, the placement and type of accents. The sentence is written as a list of strings corre­ sponding to the phonetic representation of the utterance and whose notation is compatible with Dectalk's ascii­ keyboard notation [Dectalk, 1985].

5.2 Computation for each Channel

The program runs through the set of rules for each chan­ nel computing the value of the two parameters of actions (see Fig. 2) The program retrieves all valid rules corre­ sponding to the intonational pattern defined in the input file . The chosen rules define the type and timing of oc­ currence of facial actions for each channel.

After reading in the input file, the computation of lip shapes is done (see Fig. 1). We apply speechreading technique [Jeffers and Barley, 1971] to group phonemes depending on their lip shapes. This clustering varies with the speech-rate. Coarticulation rules are then per­ formed (see section 4.3) .

To each emotion corresponds a facial expression which serves as a base with whom the other facial ac­ tions are combined. Each emotion is defined by three parameters: onset, offset (manner of appearance and disappearance of an action), and apex (time an action is stabilized).

Conversational signals are then computed. The beginning and end of an action are computed depending on the speech-rate. To follow the principle of synchrony, the program finds the closest set of phonemes whose du­ ration from the starting point and ending point of the action are equal to the onset, respectively offset, values.

When an action occurs on a pause (punctuators) its onset and offset coincide with respectively the beginning and ending of the pause. The type of movement varies with the emotion (for comma, frown will be chosen in the case of anger) or with the type of pause (period will be marked by a frown, while a question mark with a raise of the eyebrows, especially when the question is not stated linguistically) [Pelachaud, 1991b].

When a blink occurs as conversational signal or punc­ tuator (called voluntary blink) the program follows again the principle of synchrony to find its starting point. But to compute the duration of a blink, the program looks for the phonemes which have the closest timing to the average speed of a blink (it lasts about 1/4 of a sec., with 1/8 sec. of closure time, 1/24 sec. of closed eyes, and 1/12 sec. of aperture time).

Manipulators, in the present stage of this study, con­ sider only blinking (called periodic blink). Blinks should occur periodically in order to keep the eyes wet. The pro­ gram adds any necessary ones if too much time is elapsed between two consecutive voluntary blinks. We find the phonemic segment whose time from the last blink is the closest to the period of occurrence of blinks. The remain­ ing parts of the blink are synchronized with the following segments.

Regulators are mainly specified by head and eye ac­ tions. Looking toward the listener is part of a Speaker­ Turn-Signal and is emitted when the speaker wants to keep the floor . On the other hand, the speaker looks away from the listener when s/he starts speaking (i .e. Speaker-State-Signal) .

Head motions are distinguished by their amplitude and frequency. Rapid movements of the head occur on accents while pauses coincide with slowing down of head motion. After the computation of all existing head move­ ments, if no motion is specified for some phonemic ele­ ments, the head is forced to go back to its starting posi­ tion.

For a first approach, we decided to implement a sim­ pler simulation of eye movements. We supposed that when an action occurs (mutual gaze, breaking eye con­ tact, and so on) head and eyes follow the same behavior. This means we interpret eye position as head position . Otherwise, when no action is occurring, the eyes will scan the listener's face in a random way [Argyle and Cook, 1976].

6 Our Facial Model

The goal of the facial animation model we are presented in this paper is the generation of a more expressive and natural animation of the human face . The face is a com­ mon sensitive object whose movements are composed of primary actions, such as the direct muscle deformations, and secondary actions, such as the expressive wrinkles and the subtle interaction between muscles. The sec­ ondary effects of the muscle actions take part in a non negligible way in the perception of a face.

Primary deformations correspond to more visible ac­ tions: they are the bulges, the creases, and displacements of parts of the face that are involved during the contrac­ tion of a muscle. The opening of the mouth under the action of the masseter, the frowning of the eyebrows un­ der pyramidal muscles, the inflating of the cheek when laughing under the action of the masseter are easily no­ ticeable examples.

Secondary actions we are considering here and which derive from muscle contractions are of two types. The more visible ones are the expressive wrinkles. These are folds created by the skin when muscles compress it. Indeed the capacity of compression of the skin is not unlimited and varies with the age, creating rolls. The contraction of the frontal which appears during sur­ prise for example creates horizontal folds all along the forehead. We notice also as secondary actions, the sec­ ondary movements linked to the fact that muscles are not completely independent with each other. Physiolog­ ically some muscles have interleaved fibers, the action of one of the muscles can then partially act on the others. For example it is not easy to contract the upper lip and nasal wing elevator without contracting to some extent the nose wrinkler. We remark also the dependency of some symmetrical muscles: it is very difficult to frown from just one eyebrow. Some of the muscles contraction are symmetric, even if a face is never symmetric.

6.1 Generation of Primary Muscle Deformations

The muscle deformations are controlled by a dynamical physical system which computes the result of muscles acti0n in real time.

The database of the face is a spring network (2D or 3D).

The muscles are modelled by forces of different types : the shape of attraction or repulsion field created by a force on points of the spring network belonging to its zone of influence depends on its type. Six basic types are right now implemented and can be gathered to model more complex deformations. If we write P(ep the rest position of a point P; of the network, the forces on P; are:

• The forces produced by the neighbor points of the spring network 1:
• The feedback spring force of the point to its initial position J:ap
• The external forces !ext corresponding to active muscles in P;
• The viscosity force of the environment f~ = hvi

where k;j is the value of the stiffness of spring P;Pj, k; the stiffness of the feedback spring of P;, h the viscosity coefficient of the environment, vi the velocity vector of the point P;, and pype the computational function of the force depending on its type. the dynamical system is then written as:

where m; is the mass of the point P; and ai its accel­ eration. This system is solved in real time by temporal relaxation.

6.2 Generation of the Expressive Wrinkles

To the spring network representation is associated a spline representation (the knots of the spring network correspond to control points of the spline surface). 1 The expression wrinkles are creases perpendicular to the action line of the muscular force which is applied on the skin. They are geometrically modelled by creating tan­ gency discontinuities between two isolines of the spline surface.

The coefficient of skin compression is computed in function of the intrinsic age of the database, and deter­ mines the length of the curvilinear abscissa of the bulges. From the value of this curvilinear abscissa, we compute the new values of the tangents by tri-linear interpolation in a precomputed table of curvilinear abscissas based on the three parameters:

• The deviation of the tangent from its initial position around the axis of the wrinkle.
• the amplitude of the tangents.
• the distance between two control points of the two isolines that described the wrinkle.

As the distance between control points of the isolines defining a wrinkle varies with the action of muscular forces, the shape of a wrinkle varies with the muscles deformation.

6.3 Rules of Muscles Combinations

The rules of combination are given by a 2D table for which both inputs are the muscles list. The value of the links between muscles are given by a real number varying between 0 (no link) and 1 (fully linked). The user has the possibility to act in two ways on the face:

1 We have defined in an earlier publication how a database of the face can be rebuilt such that the isolines of the spline surface associated to the face are blended into the wrinkles lines of the face [Viaud and Yahia, 1992]. We will assume the spline surface has these characteristics.

• the geometric modeller ACTION3D: each muscle of the face is associated with a dial whose rotation ac­ tivates the muscle (standard input used: dial box of the Iris Silicon Graphics workstations) .
• the animation modeller ACTION3D: the user de­ fines amplitude graphs in time for each facial mus­ cle. The animation is then precomputed and a line test can visualize the result.

6.4 Facial Animation of the Speech

The AU s refer to positions. The combination rule of the AUs is the addition, i.e. it generates an average posi­ tion between the positions defined by each of the A Us. The muscles are independent physiological entities act­ ing on the face. Their combination is done by adding the amplitude. We define a table of correspondence between AU s and muscles. This conversion is made at the end of the program generating the list of movements linked to the speech. The output keyframe are then transformed into graphs for each muscle and a line test of animation is precomputed to visualize in real time the final anima­ tion.

7 Conclusion

To improve facial animation techniques, our study em­ bodies subtle facial actions as expressive elements and coordinated motions. We have presented some tools allowing the generation and animation of 3D wrinkled faces , and a method of characterizing any facial move­ ments by separating them into phonemic, intonational and emotional channels. The coordination of these var­ ious facial motions with the intonation is done by rules. This method allows us to define various speaker individ­ ualities by specifying particular sets of type and timing parameters for the facial actions and their effects on the skin (main wrinkles can be specified according to the personality of the speaker).

8 Acknowledgements

We would like to thank particularly Dr. Norman I. Badler and Dr . Mark Steedman without whom this re­ search will not have been done. We are also grateful to Steve Platt for his very useful comments. We are thank­ ful to Prof. Alberto Paoluzzi and his students for their kindness. We would like to thank also A. Gagalowicz for the interest and the enthusiasm he gives to this study.