Ubiquitous Talker: Spoken Language Interaction with Real World Objects

1 Introduction

There are many situations where we want to interact with the surrounding real world. We would also like to communicate with the objects used in our everyday life. Augmented reality is a research area that tries to incorporate an electronic information space into the real world, by means of computational devices 1 . This approach enriches, rather than replaces the real world (i.e., a virtual reality), by providing valuable informa­ tion, such as descriptions of objects, navigational help in places, and instructions for performing physical tasks.

1 The term 'augmented reality' usually refers to a variant of virtual reality, that uses see-through head-mounted dis­ plays to overlay computer-generated images on the user's real world view. For example, see [Feiner et al., 1993]. However, we use it here with a more general meaning.

Augmented reality essentially requires the ability to rec­ ognize real world objects/situations. There are several approaches to situation awareness, such as detection of physical objects using visual processing, detection of lo­ cation/orientation by positioning systems, and commu­ nication with physically embedded computers (i.e., ubiq­ uitous computing [Weiser, 1993]). For situation aware­ ness, we employ a colored barcode system [Rekimoto, 1994]. In this system, any real world object has a color­ bar tag attached to it that makes it easily identifiable.

On the other hand, in order to make natural language processing, especially spoken language processing, more practical, we must restrict or constrain the domains, contexts, or tasks, since it requires a potentially broad search space on a phonetic and linguistic level. Recently, there has been a big trend in multirnodal approaches to combine verbal and nonverbal modalities in human­ computer communication. Various sorts of nonverbal in­ formation play a role in setting the situational context, which is useful in restricting the hypothesis space con­ structed during language processing. When a context or a situation is fixed by using nonverbal information, the interpretation of utterances becomes much easier, even if the utterances are ill-formed. In other words, the correct interpretation of natural language utterances es­ sentially requires the integration of both linguistic and non-linguistic contexts. Understanding multimodal dia­ logues is not possible without some account of the role of the non-linguistic context. Considering such a con­ text, results in knowledge bases that are very efficient and robust. We have therefore introduced robust natural language processing into a system of augmented reality.

We have developed a portable system, called the Ubiq­ uitous Talker. This consists of an LCD display which presents the view at which a user is looking, as if it is a transparent glass, a CCD camera for recognizing real world objects with color-bar ID codes, a microphone for recognizing a human voice, and a speaker that outputs a synthesized voice. The Ubiquitous Talker augments re­ ality with some additional information related to a rec­ ognized object/situation. Such information is conveyed by using the LCD display and voice. The system ac­ cepts and interprets user voice requests and questions. The user may feel as if he/she is talking with the object itself through the system.

In the rest of this paper, we discuss a combination of verbal and nonverbal modalities and its role in ef­ fective interaction, explain some implementation issues and sample applications of the Ubiquitous Talker, and compare this research with some related work.

2 Situated Interaction

A real world situation includes a place where a human is a time when an event occurs, living and non-living things that exist in the vacinity, and a physical action that he or she does (e.g., looking at something).

Using situation awareness, humans can naturally in­ teract with the system by spoken language without being specially conscious of domains or regulations that con­ strain the system. This type of interaction, is called sit­ uated interaction and is very useful in certain situations. In this case, language use can be flexible and robust, since a shared situation apparently reveals a topic or fo­ cus of the dialogue and objects referred to by deictic ex­ pressions. By recognizing situations and knowing behav­ iors that humans usually do in those situations, the sys­ tem can be aware of humans' intentions and predict what they do next. Also the system can clarify humans' de­ sires by accepting information conveyed through voices and/or actions.

The semantics of natural language expressions is an­ chored to real world objects and events by means of pointing, demonstrating actions and deictic expressions such as "this," "that," "here," "there," "then," and "now." Some research on dialogue systems has combined deictic gestures and natural language such as Put-That­ There [Bolt, 1980], CUBRICON [Neal et al., 1988], and ALFRESCO [Stock, 1991].

The efficiency of situated conversation causes ambigui­ ties of conversational contents. Ambiguities are inherent in natural language communication; however, they can be resolved more easily when the real world situation is recognized, then resolved by verbally-conveyed informa­ tion alone.

In addition, the focus of attention or the focal point plays a very important role in processing applicatim~s within a broad hypothesis space, such as speech recogm­ tion. One example of focusing modality is following the humans' looking behavior. Fixation or gaze is useful for the dialogue system to determine the context of the hu­ mans' interests. For example, when a person is looking at a car, what he/she says at that time may be related to the car.

Prosodic information (e.g., voice tones) in the humans' utterances also helps to determine focus. For example, if a person puts a stress on a particular word, he/she has a special interest in an object or an event connected to the word.

Combining gestural information with spoken language comprehension shows another example of how context may be determined by the user's nonverbal behavior [Oviatt et al., 1993]. This research uses multimodal forms that prompt a user to speak or write into labeled fields. The forms are capable of guiding and segmenting inputs, of conveying the kind of information the system is expecting, and of reducing ambiguities in utterances by restricting syntactic and semantic complexities.

On the other hand, humans can move from one situa­ tion to another by physical actions (e.g., walking). When moving closer toward a situation, he/she gets informa­ tion related to the situation that is confronting him/her. This can be an intuitive process for information seeking. Walking through real world situations is a more natural way of information retrieval than a search within a com­ plex information space. Our method can be considere~ as putting a retrieval cue to a situation. For example, tf a person wants to read a book, he/she naturally has the idea of going to a place where a bookshelf exists. This means that a situation that includes a bookshelf can be a retrieval cue for searching for books.

Human memories consist of mixtures of real world sit­ uations and information that was accessed in those situ­ ations. Therefore, recognizing a real world situation can be a trigger for extracting a memory partially matched with that situation and associating information related to the memory.

In the next section, we present a prototype system based on the idea of situated interaction. The system recognizes real world situations/objects by putting ID tags on objects, then performs situated conversation. An important point is that the system accepts the real world situation as a new input modality and integrates it into spoken dialogue processing.

3 Ubiquitous Talker

The Ubiquitous Talker is a speech dialogue system with situation awareness of the real world. When it detects a real world object, interactions with it makes its user feel as if he/she is talking with the focused object itself. This is a (pseudo) portable system like so-called PDAs (Personal Digital Assistants )2.

3.1 System Configuration

Figure 1 shows the system configuration.

This system basically consists of two subsystems. One subsystem recognizes a number of real world situations that include objects with color-bar ID codes, and shows some textual and graphical information superimposed on an LCD display as shown in Figure 2 3 .

The other subsystem recognizes and interprets user speech inputs and generates voice outputs. These two subsystems communicate with each other. The image (color-code) recognizer triggers the speech recognizer and sends a message to it in order to select the appro­ priate vocabulary and grammar for analyzing the spoken utterances. It also selects a knowledge base for process­ ing the utterances and generating voice responses. The spoken message generator and the visual message (text and graphics) generator also communicate with each other and synchronize the time at which to say/ show information. The user can verbally select an item from the displayed menu, or ask some questions according to guidance conveyed by the voice and/or the text.

2 In fact, only the LCD /Camera unit is portable, and ~t is connected with a workstation. In the near future, th1s connection will be wireless.

3 This subsystem can be used independently of the speech dialogue subsystem, which is calleJ NaviCam (NAVIgation CAMera) [Rekimoto, 1995]

When the system recognizes a real world object , for instance, a calendar on the wall, the system sends a mes­ sage such as "Today is April 24, 1995. Your schedule is," and displays a timetable of the user's schedule. Then, the user asks "What about tomorrow? ," and the sys­ tem replies "Your schedule tomorrow is" and proceeds as before.

Figure 3 shows a snapshot of conversation with an object through the system.

3.2 Situation Awareness

To easily recognize real world situations/objects, we use a colored barcode system as a real world identification [Rekimoto, 1994]. The color-code consists of a number of blue and red stripes that encodes the ID of a real world object.

An image input from a small CCD camera is processed in real time by a workstation. The image recognizer continuously scans any objects with the color-code. The sys· tem generates a synthesized image by superimposing vi­ sual messages related to the color-code on the real world image obtained from the camera. The output image is shown on the LCD display. Image processing is done by software except for conversion between NTSC video signals and bitmap digital images. The output image is updated at a rate of 10 frames per second.

Figure 3: Ubiquitous Talker in Use

Recognition of objects can be naturally extended to the recognition of situations. Suppose that there is an ID code on every door in the building. When the user stands in front of a door, the system detects where the user is located and may understand what he/she intends to do by scanning the ID code on the door and by processing the information related to it.

3.3 Spoken Dialogue Processing

The speech dialogue subsystem works as follows. First, a voice input is acoustically analyzed by a built-in sound processing board. Then, a speech recognition module is invoked to output word sequences that have been as­ signed higher scores by a probabilistic phoneme model and phonetic dictionaries that are changed according to the situation.

These word sequences are syntactically and seman­ tically analyzed and disambiguated by applying a rela­ tively loose grammar and a restricted domain knowledge. Using a semantic representation of the input utterance, a plan recognition module extracts the speaker's inten­ tion. For example, from the utterance "I want to learn computer science" at a library front desk, the module interprets the speaker's intention as "The speaker wants to get information about books on computer science (for example, the place where he/she can get them)."

Once the system determines the speaker's intention, a response generation module is invoked. This gener­ ates a response to satisfy the speaker's request. Finally, the system's response is outputted as a voice by a voice synthesis module. This subsystem also sends a message to the visual message generator about what graphical and/or textual information should be displayed with the voice responses.

Speaker-independent continuous speech inputs are ac­ cepted without special hardware. To obtain a high level of accuracy, context-dependent phonetic hidden Markov models are used to construct phoneme-level hypotheses [Itou et al., 1992]. The speech recognizer outputs N­ best word-level hypotheses. As mentioned above, an ap­ propriate phonetic dictionary is dynamically selected by considering the speaker's real world situation. Therefore, the perplexities or hypothetical spaces are always main­ tained in tractable sizes without more advanced (and high-cost) speech technologies.

The semantic analyzer handles ambiguities in syntac­ tic structures and generates a semantic representation of the utterance. We applied a preferential constraint satisfaction technique for disambiguation and semantic analysis [Nagao, 1992]. For example, the following se­ mantic representation is constructed from the utterance "I want to learn computer science" at the library front desk.

(*want-1 11 *want-1 indicates that it is an instance I 1 of frame *want . (:agent *i -1) (:theme (*learn-1 (:agent *i-1) (:theme *computer-science-1))) (:situation *library-front-1) :sit~ation is added by the situation ;; awareness module.

The plan recognition module determines the speaker's intention by constructing his/her belief model and dy­ namically adjusting and expanding the model as the con­ versation progresses [Nagao, 1993]. We use a plan library that is selected according to the situation. In the case of the above example, library-front-plan is selected. Then, the recognized intention will be as follows 4 ·

4 Actually, the intention may have several candidates that

(*intend-to-know-1 1 1 *intend-to-know comes from (*want 11 (:theme *learn)) in library-front-plan. (:agent *speaker-1) ;; *i-1 is replaced with *speaker-1. (:theme (*location-of-bookshelf-1 (:area *computer-science-1))) 1 1 *location-of-bookshelf is inserted by 11 means of plan inference.

The spoken message generation module generates a response by using a domain-dependent knowledge base and text templates (typical patterns of utterances). It selects appropriate templates and combines them to con­ struct a response that satisfies the speaker's request.

3.4 Integration of Linguistic and Non-Linguistic Contexts

When the system detects a real world situation, it per­ forms not only a selection of knowledge sources (e.g. , phonetic/linguistic dictionaries) but also the introduc­ tion of a non-linguistic context. A non-linguistic context includes an object at which a user is currently looking, a location where he/she currently is, graphical informa­ tion displayed on the screen, and chronological relations of situation shifts. On the other hand, a linguistic con­ text involves semantic contents of utterances, displayed textual information, and inferred beliefs and intentions (plans and goals) of the user.

Knowing the user's intention is necessary for natural human-computer interaction. A real world situation is just a clue for it. However, integrating the non-linguistic context introduced with the situation, with the linguistic context constructed by dialogue processing, is an impor­ tant step.

In general, user's intentions are abductively inferred by using a plan library [Nagao, 1993]. A plan library is represented as an event network whose nodes are events with their preconditions and effects, and links are is-a/is­ part-of relationships [Kautz, 1990].

In the Ubiquitous Talker, plan inference is initially triggered by introducing a new non-linguistic context, since the motivation of our situated interaction is closely related to the physical actions for entering a new situ­ ation. For example, in a situation where a person is standing in front of a bookshelf, for example a bookshelf on computer science, the situation will motivate the per­ son to search for a book on computer science, read it, and study it. Therefore, when the dialogue system is aware of the situation by recognizing the bookshelf's ID, the plan library shown in Figure 4 is introduced and used for further plan inference. In this figure, the upward­ pointing thick arrows correspond to is-a (a-kind-of) rela­ tionships, while downward-pointing thin arrows indicate has-a (part-of) relationships.

Introducing and focusing a specific plan library makes plan recognition easier and more feasible.

Another connection between lin~uistic and non­ linguistic contexts is deictic center·s [Zancanaro et al.,

are assigned numerical preference values.

1993] that are possible referents of deictic expressions. The object and the location in a non-linguistic context can be current deictic centers. Also graphical and tex­ tual information on the screen includes deictic centers. Preferences on possible deictic centers as a referent are determined based on coherence of a dialogue like the case of anaphora/ ellipsis resolution in a linguistic con­ text [Walker et al., 1994].

4 Applications

We now describe some sample applications of the Ubiq­ uitous Talker 5 . The current implementation of these ap­ plications is limited, so the size of the vocabulary and the database are relatively small. However, the concept presented here is sufficiently feasible and scaleable b~ed on current technology. The examples show that situ­ ated interaction appropriately helps human activities of information seeking, and supports human everyday life.

4.1 Augmented Library

The electronic library plan is to make all published ma­ terial computerized, and to construct an efficient, easy­ to-use management and retrieval system for them (i.e., a library in virtual space). However, it is a very difficult task to digitize all published documents, and as long as physical libraries exists in the real world, we must think about the connection between physical and electronic li­ braries. In the real world library, people must be able to retrieve useful information from an electronic library by using the link between this and physical libraries.

The Ubiquitous Talker can embody electronic infor­ mation in the confronted real world situation. Therefore in a physical library, when the system is aware of the sit­ uation, it creates an information space that augments the library. This works well even if this information space is not yet mature and well constructed. For example, this information does not need to include the contents of books themselves, because books exist in the real world.

5 Actually, the system accepts and speaks only in Japanese. The example is a translation.

We call the composite space of physical and elect ronic information an a'Ugm.ented library.

When a user uses his/her Ubiquitous Talker at the li­ brary front desk (observing the signboard with the color­ bar through it), a voice comes to the user from the sys­ tem, saying "This is the library of the Tokyo Institute of Technology. Which area do you want?". The user replies "Computer science". The system then shows a map and indicates the route to the bookshelf for computer sci­ ence books before saying "Please take this route." After he/she reaches the bookshelf, he/she sees it through the system, which then says, (in fact the voice is generated by the system) "Here we have books on computer sci­ ence. What are you looking for?" After replying "A book on language," it then asks "Which kind of lan­ guage, a programming language or natural language?" Assuming the user does not know what natural language is, he/she asks "What is natural language?" The re­ sponse is "Natural language is the language that humans use for communication."

When the user asks "Where are the programming lan­ guage books?," the system replies "Books on program­ ming languages are on the third shelf of this bookshelf." He/she selects a book there and looks at it through the system. Then the book seems to say "The title of this is 'Object-oriented languages' and this was written by Mario Tokoro" and a description of the book is shown on the LCD display. The user asks "Tell me about the author," so it then displays and relates the profiles, achievements, and other publications of the author with a photograph. After seeing the publication list , the user asks "Where is the fourth book on this publication list?" The answer is "This is about computer architecture and is fifth from the right on the top shelf."

In implementing the processing described above, we developed a situation table which makes relationships between the identified situation, resources for generat­ ing messages, and related phonetic/linguistic dictionar­ ies and knowledge bases (including plan libraries) for processing utterances. Part of the situation table is shown in Table 1.

When people know the exact name of a book that they want to read and/or the name of its author before coming to the library, they will immediately ask the location of the book at the library front desk. However, it is not feasible for the system to accept speech inputs of all names of the books that the library has and the names of their authors, regardless of their areas. In this case, typing and/or hand-written inputs must be accepted by the system. In this paper, we concentrate on making speech interaction more feasible by using the information about the physical environment.

4.2 Talking Signboards

Signboards are salient markers that act as representa­ tives for certain situations or objects. People see a sign­ board when they have some interests in what is prepared beyond the signboard rather than the signboard itself. The Ubiquitous Talker plays the role of a mediator for the conversation between a human and a signboard. In this case, in order to pinpoint a user's request , the systern tries to make the situation more specific by making suggestions on the display.

For example, suppose a man stands in front of a french restaurant, e.g. 'Maxim's de Paris,' and looks at its sign­ board through his Ubiquitous Talker. The system iden­ tifies the restaurant from the ID code attached to the signboard. Then, a voice message announces "Welcome to 'Maxim's de Paris.' We are ready to tell you about the following items," and a text message appears on the LCD display. It describes "1. Menu and Price, 2. Spe­ cial Dishes recommended by the Chef, 3. Wine List, ... " When the user asks about the menu, the system displays it and says "Ok, here you are." The user can also ask for a more detail description of any dish listed on the menu.

5 Related Work

There are several other researchers following similar di­ rections to ours. The major difference between our work and that of others is that we have employed relatively light-weight and robust techniques for visual and lan­ guage processing and have combined them effectively. Below we discuss related work in more detail.

5.1 Ubiquitous Computing

Ubiquitous computing [Weiser, 1993] proposes that very small computational devices (i.e., ubiquitous computers) be embedded and integrated into physical environments in such a way that they operate seamlessly and almost transparently. These devices are aware of their physical surroundings. In contrast to ubiquitous computers, our barcode (color-code) system is a low cost and reliable so­ lution to making everything a computer. Suppose that every page in a book has a unique barcode. When the user opens a page, its page ID is detected by the system, so it can supply specific information regarding the page. When the user adds some information to the page, the system stores it with the page ID tagged for later re­ trieval. This is almost the same as having a computer in every page of the book without the cost. Our ID­ aware system is better than ubiquitous computers from the viewpoint of reliability and cost-performance, since it does not require batteries and never breaks down.

5.2 Chameleon: A Spatially-Aware Palmtop

Chameleon [Fitzmaurice, 1993] is a spatially-aware palmtop computer. It shows situated information ac­ cording to its three-dimensional location and orientation on a small LCD display. This system is not as robust be­ cause it is aware of the situation only from the location of the system itself. When physical objects change their lo­ cations, it cannot recognize their movements. The Ubiq­ uitous Talker recognizes the situation from the visual in­ formation of the real world. Therefore, it can follow the changes of the real world. In addition, since our sys­ tem integrates a speech dialogue technique, the user can interact with the system more flexibly and deeply.

5.3 Agents

Our research is motivated by not only augmented reality but also agent-oriented systems [Laurel, 1990; Nagao and Takeuchi, 1994]. Agents (or robots) recognize real world situations by using complicated perception techniques. These techniques are currently not fully developed and difficult to use in practice. We have adopted an easy, reliable situation awareness technique by using machine­ recognizable IDs. This idea is also applicable for agents.

Another key issue of agent-oriented interfaces is the inference of user's intention. Our solution is to use non-linguistic contexts for focusing the user's attention and restricting the user's possible plans. Real world situations can be information for selecting appropriate plan libraries. Therefore, our real world-oriented speech dialogue processing improves not only the accuracy of speech/language analysis, but also the tractability of in­ tention recognition.

6 Final Remarks

We have developed an augmented reality system that integrates situation awareness and spoken dialogue pro­ cessing techniques. We employed a barcode (color­ code) system for easily recognizing real world situa­ tions/objects. Situation awareness contributes to re­ ducing any hypothesis space constructed during lan­ guage processing, reducing the complexity associated with understanding speech. Situated conversation makes human-computer interaction more natural and efficient by combining linguistic and non-linguistic contexts.

As a future research direction, we plan to integrate more communication channels and modalities. For ex­ ample, detection of the user's head/eye orientation will be useful when it is necessary to determine the user's more precise focus of attention in a complex situation where there are several objects which are not placed in an ordered fashion. In addition, we are interested in the integration of prosodic information in speech recog­ nition/ synthesis.

We are also extending the technique for identifying real world situations. Currently, our color-code system is so naive that it is difficult to scale up. There are, however, some wireless electronic label systems that use batteryless passive ICs. These technologies are applica­ ble and would improve our system. In addition, location awareness methods such as the global positioning system are also useful for situation awareness, for example Fitz­ maurice's Chameleon system. Several real world infor­ mation (location, ID labels, time, distance, etc.) can in­ crease the accuracy of situation recognition and be used as sources of non-linguistic contexts.

Acknowledgments

The authors would like to thank Mario Tokoro and col­ leagues at Sony CSL for their encouragement and dis­ cussion. We also extend our thanks to Hiroko lnui and Takashi Miyata for their contributions to the implemen­ tation of the prototype system.