Pointing, Language and the Visual World: Towards Multimodal Input and Output for Natural Language Dialog Systems (Panel)

Pointing, Language and the Visual World:

Towards Multimodal Input and Output for Natural Language Dialog Systems

Introduction to a panel chaired by

Wolfgang Wahlster

Computer Science Department University of Saarbrucken

6600 Saarbrucken 11, FRG

Panelists: Jiirgen Allgayer, Erhard W. Hinrichs, Jaap Ph. Hoepelman, Willem Levelt, Norman Sondheimer

In face-to-face conversation humans frequently use deic­ tic gestures (e.g. the index finger points at something) in parallel to verbal descriptions for referent identification. Such a multimodal mode of communication can improve human interaction with machines, as it simplifies and speeds up reference to objects in a visual world.

The basic technical prerequisites for the integration of pointing and natural language (NL) are fulfilled (high­ resolution bit-mapped displays and window systems for the presentation of visual information, various pointing devices such as light-pen, mouse, and touch-sensitive screens for deictic input). But the remaining AI problem is that expli­ cit meanings must be given to natural pointing behavior in terms of a formal semantics of the visual world.

Unlike the usual semantics of mouse clicks in direct ma­ nipulation environments, in human conversation the region at which the user points (the demonstratum) is not neces­ sarily identical with the region which he intends to refer to (the referent). In conventional systems there exists a sim­ ple one-to-one mapping of a demonstratum onto a referent, and the reference resolution process does not depend on the situational context. Moreover, the user is not able to con­ trol the granularity of a pointing gesture, since the size of the predefined mouse-sensitive region specifies the granularity.

Compared to that, natural pointing behavior is much more flexible, but also possibly ambiguous or vague. With­ out a careful analysis of the discourse context of a gesture there would be a high risk of reference failure, as a deictic operation does not cause visual feedback from the referent (e.g. inverse video or blinking as in direct manipulation systems).

Although the 'common visual world' of the user and the system could be any graphics or image, current projects combining pointing and natural language focus on forms or geographic maps.

For example, the TACTILUS subcomponent of our XTRA system handles a variety of tactile gestures, inclu­ ding different granularities, inexact pointing gestures, and pars-pro-toto deixis. In the latter case, the user points at an embedded region when actually intending to refer to a su­ perordinated region. XTRA provides NL access to an ex­ pert system, which assists the user in filling out a tax form. During the dialog, the relevant page of the tax form is dis­ played on one window of the screen, so that the user can re­ fer to regions of the form by tactile gestures. The syntax and semantics of the tax form is represented as a directed acyclic graph (including relations such as 'geometrically embedded' or 'conceptual part of), which contains links to concepts in a KL-ONE knowledge base.

The deixis analyzer of XTRA is realized as a con­ straint propagation process over these networks. In addi­ tion, TACTILUS uses various other knowledge sources of XTRA (e.g. the semantics of the accompanying verbal de­ scription, case frame information, the dialog memory) for the interpretation of the pointing gesture .

While the simultaneous exploitation of both verbal and non-verbal channels provides maximal efficiency, most of the current prototypes don't use truly parallel input techni­ ques, since they combine typed NL and pointing. In these systems the user's hands move frequently back-and-forth from the keyboard to the pointing device. Note however, that multimodal input makes even NL interfaces without speech input more acceptable (less keystrokes) and that the research on typed NL forms the basis for the ultimate speech understanding system.

Another restriction of current prototypes is that the pre­ sented visual material is fixed and finite, so that the system builder can encode its semantics into the knowledge base. While some of the recent NL interfaces respond to queries by generating graphics, they are not able to analyze and ans­ wer follow-up questions about the form and content of this graphics, since they do not have an appropriate representa­ tion ofits syntax and semantics. Here one of the challenging problems is the automatic formalization of synthetic visual information as a basis for the interpretation of gestural in­ put.

Some of the open questions addressed by the panel are:

• How can non-verbally communicated information be included in a formal semantic representation of dis­ course?
• What is an adequate architecture of parsers and genera­ tors for multi modal communication?
• What effects have gestures on the attentional state and intentional structure of a dialog?
• How could a generator decide whether to use a pointing gesture, a verbal description or a combination of both for referent identification (knowledge-based media choice)?
• What are the temporal interdependences of verbal and non-verbal output in deictic expressions (synchroniza­ tion of speech and gesture)?
• How can we cope with complex pointing actions, e.g. a continuous movement of the index finger (drawing a circle around a group of objects, underlining something, specifying a direction or a path) or a quick repetition of discrete pointing acts (emphatic pointing, multiple refe­ rence)?

Part 1: The User Viewpoint

Moderator:

Bruce B. Johnson Arthur Andersen & Company

Panelists:

Dennis O'Connor Digital Equipment Corporation

Sabino Di Monte La Servitheque

Tom Bell Chevron

Paul R. Hessinger Computer Task Group, Inc.

Artificial Intelligence is being used world-wide to perform a variety of functions in business and industry. This panel will present actual user experiences with AI systems in use today, and will describe problems, expected and unexpected, that arose during the development, delivery, and maintenance of these systems. Solutions for actual problems, when they could be found, will be recounted by the panelists. Did AI techniques meet their expectations for solving these problems? If not, what should vendors of AI technology offer to better meet user needs?

PANELS

THE COMM£RCJALIZATION OF A.I.

Part 2: The Vendor Viewpoint

Moderator:

Harry Reinstein Aion Corporation

Panelists:

Tom Kehler IntelliCorp

Herbert Schoor IBM Corporation

Robert N. Goldman Artificial Intelligence Corporation

Carl Wolf Gold Hill Computers

This panel will present vendors' experiences in their attempts to offer users various products and services based on AI technologies. What do the vendors perceive to be the commercial value of different AI techniques in the marketplace? What are the primary challenges they face in gaining user acceptance? Has the marketplace for AI products changed significantly in the past few years? If so, how? Is user education still a problem, or are today's users more sophisticated in their knowledge of AI? Are vendors successfully integrating their products into existing business environments? What are the greatest obstacles still to be overcome?

PANELS: THE COMMERCIALIZATION OF A.I. Continued

Part 3: The Technology Challenge

Moderator:

·Jan Aikins Aion Corporation

Panelists:

Mike Williams lntelliCorp

Mark Fox Carnegie Group, Inc.

Jay M. Tenenbaum Schlumberger Palo Alto Research

Danny Bobrow Xerox Corporation

This panel combines suppliers of AI technology from universities, research centers, and vendors, with current users of AI technology in a discussion of challenges that remain to be solved. Can we identify the primary problem areas where researchers should devote their attention and vendors should offer new products or services? Are new knowledge representations required, new control structures, better user interfaces, or perhaps even more sophisticated logics? Should we study more carefully the marriage of AI with other emerging technologies (e.g., Computer-Aided Software Engineering) in order to offer a more complete solution to the user's problems? What needs to be done to make the AI technology user more productive and more successful? These and other questions will be discussed as this panel a ttemps to tie together issues from the user and vendor panels in a discussion of technology challenges.