Multimodal Integration A Biological View

Introduction

This paper proposes a novel perceptual architecture motivated by surprising results about how the brain processes sensory information. These results, gathered by the cognitive science community over the past 50 years, have challenged century long held notions about how the brain works and how we experience the world we live in. We argue that current approaches to building multimodal systems that perceive and interact with the real, human world are flawed and based largely upon assumptions described by Piaget (1954) - although tracing back several hundred years - that are no longer believed to be particularly accurate or relevant.

Instead, we present a biologically motivated methodology for designing interactive systems that reflects more of how the brain actually appears to process and merge sensory inputs. This draws upon neuroanatomical, psychophysical, evolutionary, and phenomenological evidence, both to separated from the specific sensory mechanisms that generate them and then integrated into higher-level representations.

This material is based upon work supported by the Advanced Research Projects Agency of the Department of Defense under contract number DAAD17-99-C-0060, monitored through Rome Laboratory. Additional support was provided by the Microsoft Corporation.

This paper examines the idea that multimodal integration is a fundamental component of perception itself and can shape individual unimodal perceptions as much as does actual sensory input. This idea is well supported biologically, and we argue here for the benefits of building interactive systems that support cross-modal influence - systems in which sensory information is shared across all levels of perceptual processing and not just in a final integrative stage. Doing this, however, requires a specialized approach that differs in basic ways from how interactive systems are generally designed today.

This paper presents our methodology for constructing multimodal systems and examines its theoretic motivation. We have followed this approach in creating the most recent version of a highly interactive environment called the Intelligent Room, and we argue that doing so has provided the Intelligent Room with unique perceptual capabilities and provides insight into building similar complex multimodal systems. We specifically examine here how the Intelligent Room's vision and language systems share information to augment each other and why the traditional ways these systems are created made this sharing so difficult to implement. Our approach is similar in spirit to the work of (Atkeson et al. 2000, Brooks et al. 1998, Cheng and Kuniyoshi 2000, Ferrell 1996, Nakagawa 1999, and Sandini et al. 1997). Although they are primarily concerned with sensorimotor coordination, there is a common biological inspiration and long-term goal to use knowledge of human and animal neurophysiology to design more sophisticated artificial systems.

Background

Who would question that our senses are distinct? We see, we feel, we hear, we smell, and we taste, and these are qualitatively such different experiences that there is no room for confusion among them. Even those affected with the peculiar syndrome synesthesia, in which real perceptions in one sense are accompanied by illusory ones in another, never lose awareness of the distinctiveness of the senses involved. Consider the woman described in (Cytowic 1988), for whom a particular taste always induced the sensation of a particular geometric object in her left hand. A strange occurrence indeed, but nonetheless, the tasting and touching - however illusory - were never confused; they were never merged into a sensation the rest of us could not comprehend, as would be the case, for example, had the subject said something tasted octagonal. Even among those affected by synesthesia, the sensory channels remain extremely well defined.

Given that our senses appear so unitary, how does the brain coordinate and combine information from different sensory modalities? This has become known as the binding problem, and the traditional assumption has been to assume that the sensory streams are abstracted, merged, and integrated in the cortex, at the highest levels of brain functioning. This was the solution proposed by Piaget (1954) and in typical Piagetian fashion, assumed a cognitive developmental process in which children slowly developed high-level mappings between innately distinct modalities through their interactions with the world. This position directly traces back to Helmholtz (1856), and even earlier, to Berkeley (1709) and Locke (1690), who believed that neonatal senses are congenitally separate and interrelated only through experience. The interrelation does not diminish the distinctiveness of the senses themselves, it merely accounts for correspondences among them based on perceived cooccurrences.

The Piagetian assumption underlies nearly all modern interactive, multimodal systems - it is the primary architectural metaphor for multimodal integration. The unfortunate consequence of this has been making integration a post-perceptual process, which assembles and integrates sensory input after the fact, in a separate mechanism from perception itself. In these multimodal systems, each perceptual component provides a generally high-level description of what it sees, hears, senses, etc. (Some systems, such as sensor networks, provide low-level feature vectors, but the distinction is not relevant here.) This, for example, may consist of locations of people and how they are gesturing, what they are saying and how (e.g., prosody data), and biometric data such as heart rates. This information is conveyed in modal-specific representations that capture detail sufficient for higher-level manipulation of the perceptions, while omitting the actual signal data and any intermediate analytic representations. Typically, the perceptual subsystems are independently developed and trained on unimodal data; each system is designed to work in isolation. (See Figure 2.) They are then interconnected through some fusive mechanism (as in Figure 1) that combines temporally proximal, abstract unimodal inputs into an integrated event model. The integration itself may be effected via a neural network (e.g., Waibel et al. 1995), hidden Markov models (e.g., Stork and Hennecke 1996), unification logics (e.g., Cohen et al. 1997), or various ad hoc techniques (e.g., Wu et al. 1999). The output of the integration process is then fed into some higher-level interpretative mechanism - the architectural equivalent of a cortex.

This post-perceptual approach to integration denies the possibility of cross-modal influence, which is pervasive in biological perception. Our visual, auditory, proprioceptive, somatosensory, and vestibular systems influence one another in a complex process from which perceptions emerge as an integrated product of a surprising diversity of components. (For surveys, see Stein and Meredith 1993, Lewkowicz and Lickliter 1994, and to a lesser extent, Thelen and Smith 1994.) For example, consider the seminal work of McGurk and MacDonald (1976). In preparing an experiment to determine how infants reconcile conflicting information in different sensory modalities, they had a lab technician dub the audio syllable /ba/ onto a video of someone saying the syllable /ga/. Much to their surprise, upon viewing the dubbed video, they repeatedly and distinctly heard the syllable /da/ (alternatively, some hear /tha/), corresponding neither to the actual audio nor video sensory input. Initial assumptions that this was due to an error on the part of the technician were easily discounted simply by shutting their eyes while watching the video; immediately, the sound changed to a /ba/. This surprising fused perception, subsequently verified in numerous redesigned experiments and now known as the McGurk effect, is robust and persists even when subjects are aware of it.

The McGurk effect is perhaps the most convincing demonstration of the intersensory nature of face-to-face spoken language and the undeniable ability of one modality to radically change perceptions in another. It has been one of many components leading to the reexamination of the Piagetian introspective approach to perception. Although it may seem reasonable to relegate intersensory processing to the cortex for the reasoning (as opposed to perceptual) processes that interested Piaget, such as in cross modal matching, it becomes far more implausible in cases where different senses impinge upon each other in ways that locally change the perceptions in the sensory apparatus themselves. One might object that the McGurk effect is pathological - it describes a perceptual phenomenon outside of ordinary experience. Only within controlled, laboratory conditions do we expect to have such grossly conflicting sensory inputs; obviously, were these signals to co-occur naturally in the real world, we would not call them conflicting. We can refute this objection both because the real world is filled with ambiguous, sometimes directly conflicting, perceptual events, and because the McGurk effect is by no means the only example of its kind. There is a large and growing body of evidence that the type of direct perceptual influence illustrated by the McGurk effect is commonplace in much of ordinary human and more generally animal perception, and it strongly makes the case that our perceptual streams are far more interwoven than conscious experience tends to make us aware. For example, the sight of someone's moving lips in an environment with significant background noise makes it easier to understand what the speaker is saying; visual cues - e.g., the sight of lips - can alter the signal-to-noise ratio of an auditory stimulus by 15-20 decibels (Sumby and Pollack 1954). Thus, a decrease in auditory acuity can be offset by increased reliance on visual input. Although the neural substrate behind this interaction is unknown, it has been determined that just the sight of moving lips - without any audio component - modifies activity in the auditory cortex (Sams et al 1991). In fact, psycholinguistic evidence has long lead to the belief that lip-read and heard speech share a degree of common processing, notwithstanding the obvious differences in their sensory channels (Dodd et al 1984).

Perhaps the most dramatic cross-modal interactions were demonstrated in the landmark studies of Meltzoff and Moore (1977), who showed that infants could imitate an investigator's facial expression within hours of birth. For example, the investigator sticking out his tongue lead the infant to do the same, although the infant had never seen its own face. Somehow, the visual cue is matched with the proprioceptive sensation of tongue protrusion. It is extraordinarily difficult to view this as a learned behavior; this will be quite relevant in considering below where the knowledge governing cross-modal influence should come from in computational systems.

We believe that the current approach to building multimodal interfaces is an artifact of how people like to build computational systems and not at all well-suited to dealing with the cross-modal interdependencies of perceptual understanding. Perception does not seem to be amenable to the clear-cut abstraction barriers that computer scientists find so valuable for solving other problems, and we claim this approach has lead to the fragility of so many multimodal systems. We also dispute the notion that perception generally corresponds well to a discrete and symbolic event model. Although such a model is well suited to many types of computational applications, it is frequently more useful to view perception as a dynamic process, rather than an event occurring at some point in time (Thelen and Smith 1994). This is all the more so during development (i.e. learning), where the space of events is fluid and subject to constant change, and during sensory fusion, where complex codependences and interactions among the different senses confound our simple event-biased predispositions. A similar dynamic approach has been taken by (Ullman 1996) for explaining cross-feature influence in visual perception.

A Multimodal System

The examples for our discussion of multimodal interactions will be drawn from the Intelligent Room project, as described in (Coen 1998, 1999). The Intelligent Room has multiple perceptual user interfaces, supporting both visual and verbal interactions, which connect with some of the ordinary human-level events going on within it. The goal of the room is to support people engaged in everyday, traditionally non-computational activity in both work and leisure contexts. Figure 3 contains a picture of the room, which contains nine video cameras, three of which the room can actively steer, several microphones, and a large number of computer-interfaced devices. Because the Intelligent Room is designed to support a range of activities that have never before explicitly involved computers, it in no way resembles a typical computer science laboratory. Its computational infrastructure is removed from view to enhance the room's appeal as a naturally interactive space with a decidedly understated "low-tech" atmosphere. The room's computational infrastructure (Coen et al. 1999), consisting of over 120 software agents running on a network of ten workstations, is housed in an adjacent laboratory.

Before exploring scenarios for new types of multimodal interactions, we first examine a traditionally explicit one in the resolution of a deictic reference, i.e., use of a word such as this in referring to a member of some class of objects. Suppose, for example, someone in the room says, "Dim this lamp." The room uses its ability to track its occupants, in conjunction with a map of its own layout, to dim the lamp closest to the speaker when the verbal command was issued. This kind of interaction can be implemented with a simple post-perceptual integration mechanism that reconciles location information obtained from the person tracker with the output of a speech recognition system. Here, multimodal integration of positional and speech information allows straightforward disambiguation of the deictic lamp reference.

Motivations

Given the simplicity of the above example, it seems far from obvious that a more complex integration mechanism is called for. To motivate a more involved treatment, we start by examining some of the problems with current approaches.

Despite many recent and significant advances, computer vision and speech understanding, along with many other perceptual interface research areas (Picard 1997, Massie and Salisbury 1994), are still infant sciences. The non-trivial perceptual components of multimodal systems are therefore never "perfect" and are subject to a wide variety of failure modes. For example, the room may "lose" people while visually tracking them due to occlusion, coincidental color matches between fore and background objects, unfavorable lighting conditions, etc. Although the particular failure modes of the modalities varies with them individually, it is a safe assumption that under a wide variety of conditions any one of them may temporarily stop working as desired. How these systems manifest this undesired operation is itself highly idiosyncratic. Some may simply provide no information, for example, a speech recognition system confused by a foreign accent. Far more troublesome are those that continue to operate as if nothing were amiss but simply provide incorrect data, such as a vision-based tracking system that mistakes a floor lamp for a person and reports that he is standing remarkably still.

That perceptual systems have a variety of failure modes is not confined to their artificial instantiations. Biological systems also display a wide range of pathological conditions, many of which are so engrained that they are difficult to notice. These include limitations in innate sensory capability, as with visual blind spots on the human retina, and limited resources while processing sensory input, as with our linguistic difficultly understanding nested embedded clauses (Miller and Chomsky 1963). Stein and Meredith (1994) argue for the evolutionary advantages of overlapping and reinforcing sensory abilities; they reduce dependence on specific environmental conditions and thereby provide clear survival advantages. A striking example of this is seen in a phenomenon known as the "facial vision" of the blind. In locating objects, blind people often have the impression of a slight touch on their forehead, cheeks, and sometimes chest, as though being touched by a fine veil or cobweb (James 1890, p204). The explanation for this extraordinary perceptory capability had long been a subject of fanciful debate. James demonstrated, by stopping up the ears of blind subjects with putty, that audition was behind this sense, which is now known to be caused by intensity, direction, and frequency shifts of reflected sounds (Arias 1996). The auditory input is so successfully represented haptically in the case of facial vision that the perceiver himself cannot identify the source of his perceptions.

Research on interactive systems has focused almost entirely on unimodal perception: the isolated analysis of auditory, linguistic, visual, haptic, or to a lesser degree biometric data. It seems to put the proverbial cart before the horse to ponder how information from different modalities can be merged while the perceptory mechanisms in the sensory channels are themselves largely unknown. Is it not paradoxical to suggest we should or even could study integration without thoroughly understanding the individual systems to be integrated? Nonetheless, that is the course taken here. We argue that while trying to understand the processing performed within individual sensory channels, we must simultaneously ask how their intermediary results and final products are merged into an integrated perceptual system. We believe that because perceptual systems within a given species coevolved to interoperate, compatibility pressures existed on their choices of internal representations and processing mechanisms. In order to explain the types of intersensory influence that have been discovered experimentally, disparate perceptual mechanisms must have some degree of overall representational and algorithmic compatibility that makes this influence possible. The approach taken here is entirely gestalt, not only from a Gibsonian (1986) perspective, but because we have no example of a complex unimodal sensory system evolving in isolation. Even the relatively simple perceptual mechanisms in paramecium (Stein and Meredith 1994, Chapter 2) and sponges (Mackie and Singla 1983) have substantial cross-sensory influences. It seems that perceptual interoperation is a prerequisite for the development of complex perceptual systems. Thus, rather than study any single perceptual system in depth - the traditional approach - we prefer to study them in breadth, by elucidating and analyzing interactions between different sensory systems.

Cross-Modal Influences

How then might cross-modal influences be used in a system like the Intelligent Room? Answering this question is a twostep process. Because the Intelligent Room is an engineered as opposed to evolved system, we first need to explicitly find potential synergies between its modalities that can be exploited. Once determined, these synergies must then somehow be engineered into the overall system, and this emerges as the primary obstacle to incorporating crossmodal influences into the Intelligent Room and more generally, to other types of interactive systems.

We begin with the following two empirical and complementary observations:

1. People tend to talk about objects they are near. (Figure 4a)

2) People tend to be near objects they talk about. (Figure 4b)

These heuristics reflect a relationship between a person's location and what that person is referring to when he speaks; knowing something about one of them provides some degree of information about the other. For example, someone walking up to a video display of a map is potentially likely to speak about the map; here, person location data can inform a speech model. Conversely, someone who speaks about a displayed map is likely to be in a position to see it; here, speech data can inform a location model. Of course, it is easy to imagine situations where these heuristics would be wrong. Nonetheless, as observations they are frequently valid and it would be reasonable to somehow incorporate influences based on them into a system like the Intelligent Room. Mechanistically, we might imagine the person tracking system exchanging information with the speech recognition system. For example, the tracking system might send data, which we will call a hint, to the speech recognition system to preferentially expect utterances involving objects the person is near, such as a map. Conversely, we can also imagine that the speech recognition system would send hints to the person tracking system to be especially observant when looking for someone in indicated sections of the room, based on what that person is referring to in his speech.

Figure 4b - People are near objects they talk about. Someone speaking about the contents of a video display is more likely to be located somewhere (delineated by the triangle) from which the display is viewable to him. Here, speech information can augment person tracking.

This seems reasonable until we try to build a system that actually incorporates these influences. There are both representational and algorithmic stumbling blocks that make this conceptually straightforward cross-modal information sharing difficult to implement. These are due not only to post-perceptual architectural integration, but also to how the perceptual subsystems (such as those in Figure 2) are themselves typically created. We first examine issues of representational compatibility, namely what interlingua is used to represent shared information, and then address how the systems could incorporate hints they receive in this interlingua into their algorithmic models.

Consider a person tracking system that provides the coordinates of people within a room in real-time, relative to some real-world origin - the system outputs the actual locations of the room's occupants. We will refer to the tracking system in the Intelligent Room as a representative example of other such systems (e.g., Wren et al. 1997, Gross et al. 2000). Its only input is a stereo video camera and its sole output are sets of (x,y,z) tuples representing occupants' centroid head coordinates, which are generated at 20Hz. Contrast this with the Intelligent Room's speech recognition system, which is based upon the Java Speech API (Sun 2001), built upon IBM's ViaVoice platform, and is typical of similar spoken language dialog systems (e.g., Zue et al. 2000). Its inputs are audio voice signals and a formal linguistic model of expected utterances, which are represented as probabilistically weighted context free grammars.

How then should these two systems exchange information? It does not seem plausible from an engineering perspective, whether in natural or artificial systems, to provide each modality with access to the internal representations of the others. Thus, we do not expect that the tracking system should know anything about linguistic models nor we do expect the language system should be skilled in spatial reasoning and representation. Even if we were to suppose the speech recognition system could somehow represent spatial coordinates, e.g. as (x,y,z) tuples, that it could communicate to the person tracking system, the example in Figure 4b above involves regions of space, not isolated point coordinates. From an external point of view, it is not obvious how the tracking system internally represents regions, presuming it even has that capability in the first place. The complementary example of how the tracking system might refer to classes of linguistic utterances, as in Figure 4a above, is similarly convoluted.

Unfortunately, even if this interlingua problem were easily solvable and the subsystems had a common language for representing information, the way most perceptual subsystems are implemented would make incorporation of cross-modal data difficult or impossible. For example, in the case of a person tracking system, the real-world body coordinates are generated via three-dimensional spatial reconstruction based on correspondences between sets of image coordinates. The various techniques for computing the reconstructed coordinates, such as neural networks or fit polynomials, are in a sense closed - once the appropriate coordinate transform has been learned, there is generally no way to bias the transformation in favor of particular points or spatial regions. Thus, there is no way to incorporate the influence, even if the systems had a common way of encoding it. Here again, the complementary situation with influencing speech recognition from a tracking system can be similarly intractable. For example, not all linguistic recognition models (e.g., bigram-based) support dynamic preferential weighting for classes of commonly themed utterances. So, even if the tracking system could somehow communicate what the speech recognition system should expect to hear, the speech recognition system might not be able to do anything useful with this information.

We see that not only are the individual modal representations incompatible, the perceptual algorithms (i.e., the contents of the sensory pipelines) are incompatible as well. This comes as no surprise given that these systems were engineered primarily for unimodal applications. Unlike natural perceptual systems within an individual species, artificial perceptual systems do not co-evolve, and therefore, have had no evolutionary pressure to force representational and algorithmic compatibility. These engineered systems are intended to be data sources feeding into other systems, such as the ones performing multimodal integration, that are intended to be data sinks. There is no reason to expect that these perceptual subsystems would or even could directly interoperate.

Designing for Interaction

Our solution was to redesign the Intelligent Room's perceptual systems with the explicit intension that they should interact with each other. Doing so required fundamental representational and algorithmic changes but has made possible subtle types of cross-modal interactions that were previously unworkable. We first detail two different categories of intersensory function and then explain how the cross-modal influences described above were implemented in the Intelligent Room.

Consider, for example, the effect of touching someone and having his head and eyes turn to determine the source of the stimulus. This is clearly an example of cross-modal influence - the position of the touch determines the foveation of the eyes - but it is fundamentally different than the interaction described in the McGurk effect above, where the influence is evidenced solely in perceptual channels. The touch scenario leads to behavioral effects that center the stimulus with respect to the body and peripheral sensory organs. The McGurk effect is an example of sensory influence within perceptual channels and has no behavioral component.

Motor influences - i.e., ones that cause attentive and orientation behaviors - are by far the more understood of the two. The primary neurological substrate behind them is the superior colliculus, a small region of the brain that produces signals that orient peripheral sensory organs based on sensory stimuli. The superior colliculus contains layered, topographic sensory and motor maps that are in register; that is, co-located positions in the real world - in the sensory case representing derived locations of perceptual inputs and in the motor case representing peripheral sensory organ motor coordinates that focus on those regions - are all essentially vertically overlapping. The actual mechanisms that use these maps to effect intersensory influence are currently unknown - variants on spreading vertical activation are suspected - but there is little doubt the maps' organization is a fundamental component of that mechanism.

Far less is known neurologically about purely semantic influences - i.e., ones that have effects confined to perceptual channels. The superior colliculus itself has been directly approachable from a research perspective because the brain has dedicated inner space, namely, the tissue of the topographic maps, to representing the outer space of the real-world; the representation is both isomorphic and perspicacious, and it has made the superior colliculus uniquely amenable to study. The perceptual as opposed to spatial representations of the senses are far more elusive and are specialized to the individual modalities and the organs that perceive them.

We have used the notion of layered topographic maps to represent both motor and semantic information in the Intelligent Room. Even though the superior colliculus has not yet been identified as a substrate in non-behavioral cross-modal influences, its extensive intra-map connections to higher cortical areas - particularly the visual cortex - may indicate its role in other types of intersensory function that are confined to perceptual channels and have no behavioral component.

Using a topographic organization, we created a new model for visually tracking people within a room (Coen and Wilson 1999). The model takes advantage of the observation that much of the information needed in humancomputer interaction is qualitative in nature. For example, it may be necessary to distinguish between a person sitting on a chair, a person standing in front of a bookcase, and a person standing in a doorway, but obtaining the actual realworld coordinates of these people is generally unimportant. In our system, locations in a room that are likely to contain people are used as reference points, as in Figure 5, to interactively train a multi-camera vision system, whose current implementation has three steerable and six fixed cameras. The system learns to combine event predictions from the multiple video streams in order to locate people at these reference points in the future. The system can also dynamically track people with the steerable cameras as they move between these locations. Because most rooms have natural attractors for human activity, such as doorways, furniture, and displays, the selection of training points is usually readily apparent from the layout of the room.

Once this topographic map is created, the tracking system activates locations on the map to correspond to its observations of people in the room. As we will see in a moment, other systems in the room can also weakly activate locations on the map, which causes the room to turn a steerable camera to view the corresponding real world location. If a person is found there as a result of orienting the camera, the system then completely activates that location on the map.

The ability to interact with topographic representations of the room is not confined to the tracking system. Once the map is learned, the room builds corresponding maps to spatially categorize events in its other sensory systems, even if they have no explicit spatial component. For example, speech recognition events are topographically organized on a map dedicated just for that purpose. As utterances are heard for which the room has spatial information (either learned or explicitly provided by its programmers), it activates locations in the speech system's topographic map, which in turn activates locations in other modalities' topographic maps via vertical spreading activation, as shown in Figure 6. Conversely, other systems can weakly activate locations on the speech system's topographic map, which causes the speech system to increase the expectation probabilities of utterances associated with that location.

Thus, activations in the map are bi-directional: perceptual events in a given modality can directly activate its map locations. Maps locations can also be activated via spreading activation from corresponding positions in other system's topographic maps, which causes a corresponding change in that modality's perceptual state - here, these spreading activations cause the secondary system to either look or listen for something. It is this bi-directional activation - through which the systems can react to intersensory stimuli - that has made possible the crossmodal influences that were presented in Figure 4.

Conclusion

This paper has described our approach to incorporating cross-modal influences into the perceptual processing of the Intelligent Room. We simultaneously argued against conventional post-perceptual integration and have motivated this position with biological evidence that unimodal perceptions are themselves integrated products of multimodal sources. Our position has allowed us to explore representational and algorithmic issues in unimodal perception that can only be approached from an integrated, multimodal perspective. It has also allowed us to investigate creating more sophisticated interactive systems by incorporating more subtle intersensory cues into the Intelligent Room. Future work is both exciting and promising.

References

1. Atkeson CG, Hale J, Pollick F, Riley M, Kotosaka S, Schaal S, Shibata T, Tevatia G, Vijayakumar S, Ude A, Kawato M: Using humanoid robots to study human behavior. IEEE Intelligent Systems, Special Issue on Humanoid Robotics, 46-56. 2000.
2. Brooks, R.A., C. Breazeal (Ferrell), R. Irie, C. Kemp, M. Marjanovic, B. Scassellati and M. Williamson, Alternate Essences of Intelligence. In Proceedings of The Fifteenth National Conference on Artificial Intelligence. (AAAI98). Madison, Wisconsin. 1998.
3. Butterworth, G. The origins of auditory-visual perception and visual proprioception in human development. In Intersensory Perception and Sensory Integration, R.D. Walk and L.H. Pick, Jr. (Eds.) New York. Plenum. 1981.
4. Cheng, G., and Kuniyoshi, Y. Complex Continuous Meaningful Humanoid Interaction: A Multi Sensory-Cue Based Approach Proc. of IEEE International Conference on Robotics and Automation (ICRA 2000), pp.2235-2242, San Francisco, USA, April 24-28, 2000.
5. Coen, M. Design Principles for Intelligent Environments. In Proceedings of The Fifteenth National Conference on Artificial Intelligence. (AAAI98). Madison, Wisconsin. 1998.
6. Coen, M. The Future Of Human-Computer Interaction or How I learned to stop worrying and love My Intelligent Room. IEEE Intelligent Systems. March/April. 1999.
7. Coen, M., and Wilson, K. Learning Spatial Event Models from Multiple-Camera Perspectives in an Intelligent Room. In Proceedings of MANSE'99. Dublin, Ireland. 1999.
8. Coen, M., Phillips, B., Warshawsky, N., Weisman, L., Peters, S., Gajos, K., and Finin, P. Meeting the computational needs of intelligent
9. environments: The Metaglue System. In Proceedings of MANSE'99. Dublin, Ireland. 1999.
10. Cohen, P. R., Johnston, M., McGee, D., Smith, I. Oviatt, S., Pittman, J., Chen, L., and Clow, J. QuickSet: Multimodal interaction for simulation set-up and control. 1997, Proceedings of the Applied Natural Language Conference, Association for Computational Linguistics. 1997.
11. Cytowic, R. E., Synesthesia: A Union of Senses. New York. Springer-Verlag. 1989.
12. Darrell, T., Gordon, G., Harville, M., and Woodfill, J., Integrated person tracking using stereo, color, and pattern detection, Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR '98), pp. 601-609, Santa Barbara, June 1998.
13. Ferrell, C. Orientation behavior using registered topographic maps. In Proceedings of the Fourth International Conference on Simulation of Adaptive Behavior (SAB-96). Society of Adaptive Behavior. 1996.
14. Gross, R., Yang, J., and Waibel, A. Face Recognition in a meeting room. Fourth IEEE International Conference on Automatic Face and Gesture Recognition, Grenoble, France, March 2000
15. Held, R. Shifts in binaural localization after prolonged exposures to atypical combinations of stimuli. Am. J. Psychol. 68L526-266. 1955.
16. Helmholtz, H. v. Handbook of Physiological Optics. 1856. as reprinted. in James P.C. Southall. (Ed.) 2000.
17. James, H. 1890. Principles of Psychology. Vol. 2. Dover. 1955.
18. Kohler, I. The formation and transformation of the perceptual world. Psychological Issues 3(4):1-173. 1964.
19. McGurk, H., and MacDonald, J. Hearing lips and seeing voices. Nature. 264:746-748. 1976.
20. Meltzoff, A.N. and Moore, M.K. Imitation of facial and manual gestures by human neonates. Science 198:75-78. 1977.
21. Miller, George and Noam Chomsky (1963). Finitary models of language users. In Luce, R.; Bush, R. and Galanter, E. (eds.) Handbook of Mathematical Psychology, Vol 2. New York: Wiley. 419-93.
22. Piaget, J. Construction of reality in the child, London: Routledge & Kegan Paul, 1954.
23. Sams, M., Aulanko, R., Hamalainen, M., Hari, R., Lounasmaa, O., Lu, S., and Simola, J. Seeing speech: Visual information from lip movements modified activity in the human auditory cortex. Neurosci. Lett. 127:141-145. 1991.
24. Sandini G., Metta G. and Konczak J. "Human Sensori-motor Development and Artificial Systems". In: AIR&IHAS '97, Japan. 1997.
25. Stein, B., and Meredith, M. A. The Merging of the Senses. Cambridge, MA. MIT Press. 1994.
26. Stork, D.G., and Hennecke, M. Speechreading: An overview of image processing, feature extraction, sensory integration and pattern recognition techniques", Proc. of the Second Int. Conf. on Auto. Face and Gesture Recog. Killington, VT pp. xvi--xxvi 1996.
27. Sumby, W.H., and Pollack, I. Visual contribution to speech intelligibility in noise. J. Acoust. Soc. Am. 26:212-215. 1954.
28. Sun. http://www.javasoft.com/products/java-media/speech/. 2001.
29. Thelen, E., and Smith, L. A Dynamic Systems Approach to the Development of Cognition and Action. Cambridge, MIT Press. 1998.
30. Ullman, Shimon. High-level vision: object recognition and visual cognition. Cambridge. MIT Press. 1996.
31. Waibel, A., Vo, M.T., Duchnowski, P., and Manke, S. Multimodal Interfaces. Artificial Intelligence Review. 10:3-4. p299-319. 1996.
32. Wren, C., Azarbayejani, A., Darrell, T., and Pentland, P., "Pfinder: Real-Time Tracking of the Human Body ", IEEE Transactions on Pattern Analysis and Machine Intelligence, July 1997.
33. Wu, Lizhong, Oviatt, Sharon L., Cohen, Philip R., Multimodal Integration -- A Statistical View, IEEE Transactions on Multimedia, Vol. 1, No. 4, December 1999, pp. 334-341.
34. Zue, V., Seneff, S., Glass, J., Polifroni, J., Pao, C., Hazen, T., and Hetherington, L. "JUPITER: A Telephone-Based Conversational Interface for Weather Information," IEEE Transactions on Speech and Audio Processing, Vol. 8, No. 1, January 2000.

Real-Time Auditory and Visual Multiple-Object Tracking for Humanoids

Kazuhiro Nakadai DD , Ken-ichi Hidai DD , Hiroshi Mizoguchi DE , Hiroshi G. Okuno DDDC , and Hiroaki Kitano DDDF

DD Kitano Symbiotic Systems Project, ERATO, Japan Science and Technology Corp.

Mansion 31 Suite 6A, 6-31-15 Jingumae, Shibuya-ku, Tokyo 150-0001, Japan

DE Department of Information and Computer Science, Saitama University, Saitama 338-8570, Japan

DC Department of Intelligence Science and Technology, Kyoto University, Kyoto 606-8501, Japan

• DF Sony Computer Science Laboratories, Inc., Tokyo 141-0022, Japan

CUnakadai, hidai, okuno, kitanoCV@symbio.jst.go.jp, hm@me.ics.saitama-u.ac.jp

Abstract

This paper presents a real-time auditory and visual tracking of multiple objects for humanoid under real-world environments. Real-time processing is crucial for sensorimotor tasks in tracking, and multiple-object tracking is crucial for real-world applications. Multiple sound source tracking needs perception of a mixture of sounds and cancellation of motor noises caused by body movements. However its real-time processing has not been reported yet. Real-time tracking is attained by fusing information obtained by sound source localization, multiple face recognition, speaker tracking, focus of attention control, and motor control. Auditory streams with sound source direction are extracted by active audition system with motor noise cancellation capability from 48KHz sampling sounds. Visual streams with face ID and 3D-position are extracted by combining skincolor extraction, correlation-based matching, and multiple-scale image generation from a single camera. These auditory and visual streams are associated by comparing the spatial location, and associated streams are used to control focus of attention. Auditory, visual, and association processing are performed asynchronously on different PC's connected by TCP/IP network. The resulting system implemented on an upper-torso humanoid can track multiple objects with the delay of 200msec, which is forced by visual tracking and network latency.

1 Introduction

Humanoids and entertainment robots or at least mobile robots have attracted a lot of attention last year, e.g., at IEEE/RSJ First Humanoids-2000 conference, and are expected to play a role of human partners in the 21st century. Let us imagine the situation autonomous robots are used in social and home environment, such as a pet robot at living room, a service robot for office, or a robot serving people at a party. The robot shall identify people in the room, pay attention to their voice and look at them to identify visually, and associate voice and visual images, so that highly robust event identification can be accomplished. These are minimum requirements for social interaction [Brooks et al., 1998].

Some robots are equipped with improved robot-human interface. Jijo-2 [Asoh et al., 1997] can recognize a phrase command by speech-recognition system; AMELLA [Waldherr et al., 1998] can recognize pose and motion gestures. Kismet of MIT AI Lab [Breazeal and Scassellati, 1999] can recognize speeches by speech-recognition system and express various kinds of sensation. Hadaly of Waseda University [Matsusaka et al., 1999] can localize the speaker as well as recognize speeches by speech-recognition system.

However, the technologies developed so far are still immature; in particular, auditory processing and integrated perception among vision, audition, and motor control. At robotic conferences such as IROS, SMC, and ICRA as well as AIrelated conferences, there were at most one or two papers related to auditory processing 1 , and most papers on robot perception is limited to vision-only and vision with ultrasonic, infra-red or laser range finders. This is unfortunate because integrated processing of auditory and visual processing combined with appropriate motor control is essential in social interaction of robot systems.

For auditory and visual tracking, Nakadai et al. presented the active audition for humanoids to improve sound source tracking by integrating audition, vision, and motor controls [Nakadai et al., 2000]. An active audition system is implemented in a upper-torso humanoid to demonstrate that the humanoid actively moves its head to improve localization by aligning microphones orthogonal to the sound source and by capturing the possible sound sources by vision. Although such an active head movement inevitably creates motor noise, the system adaptively cancels motor noise using motor control signals. The experimental result demonstrates that the active audition by integration of audition, vision, and motor control enables sound source tracking in variety of conditions. One of crucial problems is a lack of real-time processing.

Matsuyama et al. presented an architecture for asynchronous coordination of sensorimotor control [Matsuyama et al., 2000] so that the camera moves smoothly to track the object in real-time. This architecture, called dynamic memory, is general, but they use it only for vision-based motor control. Shafer et al. presented the software architecture of sensor fusion for an autonomous mobile robot [Shafer et al., 1986]. The architecture is based on a parallel blackboard system, and the sensors include vision, range finder, but not microphones. It exploits global consistency regarding position and orientation of the vehicle and sensors. Murphy presented the sensor fusion system for mobile robots called the Sensor Fusion Effects (SFX) architecture, which is based on the uncertainty management system by Dempster-Shafer theory [Murphy, 1998].

1 Auditory processing shall be distinguished from speech recognition. Auditory processing (auditory scene analysis) aims at separation and understanding of multiple sound streams, such as human speech, environmental noise, music, etc.

Other robots with microphones as ears for sound source localization or sound source separation have attained little in auditory tracking. Kismet has a pair of omni-directional microphones outside the simplified pinnae [Breazeal and Scassellati, 1999]. Since it is designed for one-to-one communication and its research focuses on social interaction based on visual attention, the auditory tracking has not been implemented so far. Hadaly uses a microphone array to perform sound source localization, but the microphone array is mounted in the body and its absolute position is fixed during head movements [Matsusaka et al., 1999]. In the both cases, sound source separation is not exploited and a microphone for speech recognition is attached to the speaker.

In the research of computational auditory scene analysis (CASA) to understand a mixture of sounds, real-time processing is one of the main problems in applying CASA to real-world applications [Rosenthal and Okuno, 1998]. Realtime processing is important to take appropriate actions in daily environments where many people, robots and objects exist. Auditory and visual tracking by Nakadai et al. accepts sounds in daily environment, but does not run in real-time [Nakadai et al., 2000]. Another auditory and visual tracking by Nakagawa et al. does not run in real-time [Nakagawa et al., 1999].

Two major issues that have not been done in the past are attacked in this paper, that is, association of multiple auditory and visual streams, and real-time processing of integrated auditory and visual scene analysis.

The rest of the paper is organized as follows: Section 2 explains the robot hardware which is used as a testbed and presents the issues in real-time tracking. Section 3 describes the design of the system and the details of each module are described in Section 4. Section 5 demonstrates and evaluates the performance of the system. Section 6 discusses the observations of the experiments and future work and concludes the paper.

2 Issues in Real-Time Tracking

2.1 Robot Hardware

As a testbed of real-time multiple-object tracking, we use a upper-torso humanoid called SIG shown in Fig. 1 [Nakadai et al., 2000]. The cover of the mechanics is made of FRP and discriminates internal and external world acoustically. SIG has two microphones at the left and right ear positions to capture external sounds from outside of the body, and two microphones within the body to capture internal sounds mainly caused by motor movements. All the microphones are omnidirectional microphones of Sony ECM-77S. SIG's body has four DOFs (degree of freedom), each of which is a DC motor controlled by a potentiometer. SIG is equipped with a pair of CCD cameras of Sony EVI-G20, but the current vision module uses only one camera.

2.2 Task and Issues

The task in this paper is to track multiple objects in real-time with two kinds of sensors, a camera and two microphones, and one actuator to rotate the body. Some important issues in this task are:

• AF Robustness in visual stream extraction (temporal sequences of face localization and face identification) against non-uniform environments due to lighting conditions or moving humans.
• AF Robustness in auditory stream extraction (temporal sequence of sound source localization and sound source separation) against dynamic environments because objects (people) move and the humanoid also moves.
• AF Association of visual and auditory streams by face identification and sound source localization to compensate missing or ambiguous data. Common representation for both auditory and visual feature extractions is needed.
• AF Focus of attention control based on association to control actuators.
• AF Trade-off of processing speed vs quality of feature extractions for real-time processing.
• AF Method of synchronization between asynchronous auditory and visual processing that have different processing speeds. The frame rate of vision is 30 Hz, while the sampling rate of sound is 48KHz. Therefore, asynchronous processing is essential to exploit the full range of concurrency.

3 Design of the System

From the viewpoint of functionality, the whole system can be decomposed into five layers - SIG Device Layer, Process Layer, Feature Layer, Event Layer, and Stream Layer

(see Fig. 3) . From the viewpoint of implementation, the whole system consists of six asynchronous modules - Audition, Vision, Association, Focus-of-Attention, Motor Control, and Viewer. This relation between two viewpoints is depicted in Fig. 2. Since Vision module utilizes the full power of Pentium-III 733MHz CPU, the whole system is organized in the form of distributed processing with three Linux nodes based on Pentium-III with RedHat Linux 6.2J. The first node with 600MHz is for Audition, the second node with 733MHz for Vision, and the third with 450MHz for the rest. They are connected by TCP/IP over Fast Ethernet 100Base-TX.

The estimated processing time of each module executed on a node is summarized below:

• AF Vision - 200msec for face localization and identification,
• AF Audition-40msec for sound source localization,
• AF Motor Control - 100msec
• AF Network latency - up to 200msec

Therefore, we set the goal that the response time of the system should be 200msec of delay.

Audition and Vision generate an event by feature extraction and organize a stream as a temporal sequence of events. Motor Control also generates an event of motion. Association fuses these events to make a higher level representation. This fusion associates auditory and visual streams to make an associated stream. Focus-of-Attention makes a planning of SIG's movement based on the status of streams, that is, whether they are associated or not.

Motor Control is activated by Focus-of-Attention module and generates PWM (Pulse Width Modulation) signals to DC motors. It also sends a motor event consisting of motor direction (azimuth of the midsagittal plain) to Association module. Viewer shows the status of auditory, visual and associated streams in the radar and scrolling windows (see screen shots shown in Fig. 5). Some modules are explained in details in the next section.

4 Details of EachModule

4.1 Active AuditionModule

Sound localization for a robot or an embedded system is usually solved by using interaural phase difference (IPD) and interaural intensity difference (IID). These values are calculated by using Head-Related Transfer Function (HRTF). However, HRTF depends on the shape of head and it also changes as environments change. For real-world applications, sound localization without HRTF is preferable. Nakadai et al. proposed the method based on the auditory epipolar geometry, an extension of epipolar geometry in stereo vision to audition [Nakadai et al., 2000]. They also proposed active audition for sensorimotor task with canceling motor and mechanical noises. However, they failed in doing the jobs in real-time, because they stuck to pure-tone processing. In this paper, we extend their approach (1) by exploiting the harmonic structure to extract peaks precisely and (2) by solving the uncertainty in sound source localization by Dempster-Shafer theory.

Audition module equipped with active audition is depicted in Fig. 3. The input signal, a mixture of sounds originating from different directions, is sampled with sampling frequency of 48KHz and 16-bit quantization, and its spectrogram is calculated by Fast Fourier Transforms (FFT). Audition extracts pitches (fundamental frequency, BYBC), separates and localizes sound sources.

Peak Extraction and Sound Source Separation: First a peak is extracted by a band-pass filter, which passes a frequency between 90Hz and 3KHz if its power is a local maximum and more than the threshold. This threshold is automatically determined by the stable auditory conditions of the room. Then, extracted peaks are clustered according to harmonicity. A frequency of BYD2 is grouped as an overtone (integer multiple) of BYBC if the relation CY BYD2 BYBC A0 CQ BYD2 BYBC CRCY AK BCBMBCBI holds. The constant, 0.06, is determined by trial and error. By applying Inverse FFT to a set of peaks in harmonicity, a harmonic sound is separated from a mixture of sounds.

Sound Source Localization: Once a harmonic structure is obtained, the direction of sound source is calculated by hypothetical reasoning for IPD (Interaural Phase Difference) and IID (Interaural Intensity Difference). The azimuth (horizontal direction) is quantized and represented by every BH Æ discrete value in the range of A6BLBC Æ . The front direction of SIG is 0 Æ .

From the extracted harmonic structure of left and right channels, a pair of harmonic structures is obtained. Then the IPD,  D7 , is calculated. Auditory Epipolar Geometry generates a hypothesis of IPD  CW for each BH Æ candidate, AI [Nakadai et al., 2001]. Since the IPD is ambiguous for frequencies of more than 1200Hz, the distance, CSB4AIB5, in IPD between the data and a hypothesis is defined as follows:

Where D2 CUBOBDBEBCBCDE is the number of overtones of which frequency is less than 1200Hz.

The similar relation may hold for IID, but our experience with IID proves that it can discriminate at most the side, that is, left or right. Suppose that
D7 B4CUB5 is the IID for peak frequency CU. If the value of
BP  3000Hz CUBP1200Hz
D7 B4CUB5 is nonnegative, the direction is decides as left, otherwise as right:

Integration of IPD and IID by Dempster-Shafer theory To determine the sound source direction, the belief factors of IPD and IID are calculated and then integrated by DempsterShafer theory. The belief factor of IPD, BUBY
BW , is calculated by using probability density function defined by Eq. (2).

where D1 and D7 are the average and variance of CSB4AIB5, respectively. D2 is the number of CS.

The belief factor of IID, BUBY

BW B4AIB5 is defined by Table 1. Then, belief factors of IPD and IID, BUBY
BW and BUBY

BW , are integrated using Dempster-Shafer theory as defined in Eq. (3).

AI for the maximum BUBY
BWB7

BW is treated as the sound source direction of the harmonics. Finally, Audition sends an auditory event consisting of pitch (BYBC) and a list of 20-best directions (AI) with reliability factor for each harmonics.

4.2 Real-Time Multiple Face Tracking

Multiple face detection and identification suffers more severely from frequent changes in the size, direction and brightness of face. To cope with this problem, Hidai et al. combines skin-color extraction, correlation based matching, and multiple scale images generation [Hidai et al., 2000].

The requirements on multiple face tracking are the capability of discriminating face data of the same face ID from others and on-line learning. The first requirement is a class concept. Turk et al. proposed the eigenface matching technique as a kind of subspace method [Turk and Pentland, 1991]. A subspace for discrimination is created by Principal Component Analysis (PCA). PCA, however, does not provide the means to group a data according to its face ID, since such an ID cannot be generated by PCA. Thus, the subspace obtained by PCA is not always suitable to distinguish such classes.

On the other hand, Liner Discriminant Analysis (LDA) can create an optimal subspace to distinguish classes. Therefore, we use Online LDA [Hiraoka et al., 2000]. In addition, this method continuously updates a subspace on demand with a small amount of computation.

The face identification module (see Fig. 2) projects each extracted face into the discrimination space, and calculates its distance CS to each registered face. Since this distance depends on the degree (, the number of registered faces) of discrimination space, it is converted to a parameter-independent probability  DA as follows.

The face localization module converts a face position in 2-D image plane into 3-D world coordinate. Suppose that a face is DB A2 DB pixels located in (DC, DD) in the image plane, whose width and height are CG and CH , respectively (see screen shots shown in Fig. 5). Then the face position in the world coordinate is obtained in terms of distance D6, azimuth AI and elevation AU by the following equations.

where BV BD and BV BE are constants defined by the size of the image plane and the image angle of the camera.

Finally, Vision module sends a visual event consisting of a list of 5-best Face ID (Name) with its reliability and position (distance D6, azimuth AI and elevation AU) for each face.

4.3 Stream Formation and Association

Association module forms auditory streams from auditory events and visual streams from visual events, and associates a pair of auditory and visual streams to create a higher level stream, which is called an associated stream (see Fig. 2). The flow of processing in stream formation and association is summarized as follows (Figs. 4(a)-(d)):

1. Events from Audition, Vision and Motor modules are stored in the short-termmemory.
2. Direction information of events is converted into the absolute coordinate to treat them in the common coordinate.
3. Events are grouped into an auditory or a visual stream according to a temporal sequence of events.
4. Streams are synchronized by every 100msec to calculate the distance between streams.
5. An auditory and a visual stream which are close for more than a constant time are associated as an associated stream.

First, events are stored in the short-term memory and kept only for 2 seconds to attain incremental and real-time processing. In Fig. 4(a), where SAD, VAD and MAD represent events created by Audition, Vision and Motor modules, respectively. Each module creates events at its own cycle, e.g. 40msec for audition, 200msec for vision and 100msec for motion. Then, motor events are synchronized with auditory and visual events. To put it concretely, a motor direction when an auditory or a visual event appeared is estimated from motor events in the short-term memory. A motor event with the estimated motor direction is shown as M . Because visual events from Vision module and auditory events from Audition module are represented in robot coordinate, the directions of these events are converted to ones in the absolute coordinate by using estimated motor direction. These are represented as S or V in Fig. 4(a). This synchronization process runs with a delay of 200msec as mentioned in Sec. 3.

Auditory and visual streams are formed in Fig. 4(b). Xaxis indicates elapsed time from right to left, and Y-axis indicates azimuth in the absolute coordinate. Thin lines with small filled circles and a thick line with small rectangles represent auditory streams and a single visual stream. An auditory event is connected to the nearest auditory stream within the range of A6BDBC Æ and with common BYBC. A visual event is connected to the nearest visual stream within 40 cm and with a common face ID. In either case, if there are multiple candidates, the most reliable one is selected. If any appropriate stream is found, such an event becomes a new stream. In case that no event is connected to an existing stream, such a stream remains alive for up to 500msec. The system cannot detect an auditory or a visual event when a person stops talking and looks away for a moment. This margin of 500msec is prepared to continue streams in case of the missing event extraction. After 500msec of keep-alive state, the stream terminates.

When the distance between an auditory and a visual stream is close for more than a constant time, they are regarded as streams originating from the same object and integrated into an associated stream, which is a higher layer representation of a stream shown in Fig. 2. Because auditory and visual streams consist of events with 40msec and 200msec cycles, respectively, it is difficult to evaluate the distance between these two streams without synchronization. Then, they are synchronized with the same cycle, 100msec. Fig. 4(c) illustrates synchronized streams as lines with large circles and rectangles. If an event is not available in this case, linear regression is used for interpolation in the same way as synchronization with motor events.

An auditory and a visual streams are associated if their direction difference is within the range of A6BDBC Æ and this situation continues for more than 50% of the 1 sec period shown in Fig. 4(d).

The visual direction is usually used for the direction of the associated stream because visual information is more accurate. However, when a tracking person is occluded, the system cannot use visual information. In this case, auditory information is used for the associated stream. This suggests an advantage of integration of audition and vision, i.e. auditory information is efficient not only for pre-attentive uses such as a trigger of attention but also for compensations of missing or ambiguous information as this case. If either auditory or visual event has not been found for more than 3 sec, such an associated stream is deassociated and only existing auditory or visual stream remains. If the auditory and visual direction difference has been more than BFBC Æ for 3 sec, such an associated stream is deassociated to two separate streams.

4.4 Focus of Attention Control

SIG should pay attention for sounds from unseen objects to get further information. When such a sound does not exist, faces with sound, i.e. talking people, should have high priority because they are attractive even for human perception. The principle of focus-of-attention control hereby is as follows:

1. An auditory stream has the highest priority,
2. an associated stream has the second priority, and
3. a visual stream has the third priority.

The algorithm of focus-of-attention control is sketched by using an example shown in Fig. 5, which depicts how auditory and visual streams are generated and associated.

1. Focus of attention changes to a new association stream. [D8 BD and D8 BK of Fig. 5].
2. If one of the visual and auditory stream of an associated stream terminates due to occlusion, disappearance, or end of speech, association continues [D8 BG to D8 BH of Fig. 5].
3. If this state continues for a particular time, say 3 seconds, the focus of attention may change.
4. (a) Focus of attention changes to one of associated streams.
5. (b) If no associated stream is found, focus of attention changes to one of auditory streams. [D8 BI of Fig. 5].
6. (c) Otherwise, focus of attention changes to one of visual streams.
7. In turning the body to associate the auditory stream to visual one, focus of attention keeps the same even if a new associated stream is generated.

5 Experiments and Evaluation

A 40-second scenario shown in Fig. 5 is used as a benchmark. The performance of integrated auditory and visual tracking is shown in Fig. 5, which shows that focus of attention changes twice. In the first half of the scenario up to D8 BP BEBIsec, two speakers are apart, while in the second half they are close and viewed in the same camera view field. In both cases, the system can track the speakers well.

The direction of SIG's body is depicted in Fig. 6, which shows that the motor control succeeds in giving correct PWM motor commands. To sum up, sensorimotor task in singleand multi-speaker tracking is well accomplished.

The performance of visual tracking is shown in Fig. 7. This timechart is generated by collecting the first candidate from the internal states of Vision module. Therefore, the motor movement is the same as the above. In the first half of the scenario, occlusion causes a gap of visual streams between D8 BG and D8 BH . From D8 BI to D8 BJ , no person can be seen due to the limited range of camera view field. Fig. 5 proves that occlusion and out-of-sight can be easily recovered by associated streams.

The performance of auditory tracking is shown in Fig. 8, which is generated in the same manner as Fig. 7. The Audition module can separate two auditory streams correctly from D8 BF to D8 BP BEBFsec, and D8 BL to D8 BDBC , but generate erroneous streams around D8 BK and D8 BL . In addition, the directions of two speakers are not so correct from D8 BP BDBD sec (D8 BH ) to D8 BP BDBJ sec, because Mr. A moves and SIG tracks him by rotating its body. That is, the reverberation (echo) due to this moving talker and motor noise deteriorates the quality of sound source localization.

5.1 Limitations on the Proposed System

The room used in this experiment is about BFD1 in width and length and BED1 in height, and sound absorbing materials are attached on walls, ceiling and floor. It is not anechoic, but has reverberation time of 0.1 sec. Because the value in a normal speech studio of the equivalent size is about 0.2 sec, the room has less reverberation. Acoustic conditions, however, depends on objects in the room. When we put a plastic partition of 1.5m A2 1.5m with strong reverberation in the room, the correctness of sound source localization is reduced remarkably.

The background noise level of the room is 30 dBA on average. This is measured with the filter by A weighting, similar to human auditory characteristic. The value of 30 dBA corresponds to the noise level of a room in a quiet residence. We confirmed that the current system works well up to 40dBA of background noise level, but we have not checked above 40 dBA.

The room has six halogen lights with adjustment function of the intensity on the ceiling. The range of the light intensity is from 0 to 50 lux. Because the light intensity from 300 to 1500 lux is recommended for a normal office by JIS (Japanese Industrial Standard), even the maximum light intensity of the room is weak. Our face extraction and recognition method works well under the condition of more than 5 lux. Visual disappears.

Figure 8: Temporal Sequence of Auditory Tracking of Two Speakers in the same scenario as Fig. 5

processing is robust against the change of light intensity. In this experiment, the light intensity in the room is 50 lux.

Other benchmarks such as crossing of moving talkers, moving talkers without seeing any talkers, and alternative talking of four speakers prove that the resulting system succeeds in real-time sensorimotor tasks of tracking 2 .

2 Since the work is related to the real-time processing, the readers may be suggested to visit the following Web site: http://www.symbio.jst.go.jp/SIG/

6 Conclusion and Future Work

The key idea of real-time tracking is "For each processing, take it easy, and ambiguities will be resolved with the help of others." This idea is obtained by the scrutiny of the behavior of each component of implementations of Nakadai et al.'s work [Nakadai et al., 2000]. We do not stick to pure tones, but utilize the collective behavior of harmonic sounds; we prefer frequency resolution over the time resolution by increasing the points of FFT. We give up the precise face localization and identification. Instead, we associate auditory, visual, and motor direction information to localize the sound sources.

Some technical future work includes learning the adaptive association of different or dynamic environments. Since lighting conditions and reverberation (echo) change drastically in such environments, Vision and Audition modules should adjust their parameters on demand. In addition, Association should adapt its parameters for stream forming and association. Baysian algorithm for resolving ambiguities in stream forming and association is a promising technique. Another future work is incorporating stereo vision. Even in a static environment robust auditory processing such as sound source separation and localization would be useful when a room is noisier, has objects with strong reverberation, or has many people.

We believe that our result would open a new era of sound processing, in particular, cocktail party computer or "Shotoku-Taishi" computer that can listen to several things at once.

Auditory and visual tracking should be incorporated in a total system with robot-human interface. We have already built such a system comprising speech recognition, speaker identification, and speech synthesis based on the proposed system. Once the application is fixed, the top-down stream separation may be exploited. Some information that forces top-down stream separation includes speaker identification. The speaker information may reduce the search space of face recognition and speech recognition. For example, let us consider that crossing of two talking persons. In this case, the system may miss judging that they are approaching and then receding because speaker IDs are lacking. Thus, speaker identification can reduce this kind of ambiguity. Its design and implementation will be reported by a separate paper, since this paper focuses on the real-time processing.

Acknowledgments

We thank our colleagues of Symbiotic Intelligence Group, Kitano Symbiotic Systems Project; Mr. Tatsuya Matsui and Dr. Tino Lourens, and Prof. H. Ishiguro of Wakayama University for their discussions.