A new generation of multimodal systems is emerging in which the user is able to employ natural communication modalities, including spoken language and pen-based gesture, in addition to the usual graphical user interface technologies. To build such systems, we adopt the principle of using the strengths of one modality to overcome weaknesses in another. We discuss how to design multimodal systems according to this principle, and how to build robust multimodal architectures that employ it at runtime in a unification-based framework. These design and architectural principles will be illustrated through QuickSet — a handheld, collaborative, multimodal system that allows continuous speech and pen-based gesturing as input. QuickSet uses a fault-tolerant distributed agent architecture, runs on PC’s, and is scalable from wearable to wall-sized systems. To assess the impact of multimodal interaction, we will describe a study comparing the use of a map-based graphical user interface and multimodal interaction. After discussing reasons why graphical user interfaces fail to satisfy users in high stress environments, we present a new version of the QuickSet technology that attempts to support them through a tangible multi-modal user interface. Finally, we will discuss the challenges that await researchers when we try to support multimodal interaction among people.
Philip R. Cohen received his B.A. degree in mathematics from Cornell University, and his M.Sc. and Ph.D. degrees in computer science from the University of Toronto. Dr. Cohen has been a researcher or faculty member at Bolt Beranek and Newman, Inc., the Oregon State University, the University of Illinois, Fairchild Laboratory for Artificial Intelligence Research, and SRI International. He is currently Professor and Co-Director of the Center for Human-Computer Communication in the Dept. of Computer Science and Engineering at the Oregon Graduate Institute. His research interests include multimodal interaction, multiagent systems, dialogue, natural language processing, and theories of collaboration and communication. Cohen is currently Past President of the Association for Computational Linguistics, and is a Fellow of the American Association for Artificial Intelligence.
Halpern discusses a new formalism for reasoning about uncertainty called plausibility. Plausibility is a generalization of probability: the plausibility of a set is just an element of some arbitrary partial order (instead of being an element of [0,1], as in the case of probability). Halpern shows that plausibility can be used to give insight into belief and belief change, default reasoning, decision rules, and (if time permits) when the technology of Bayesian networks can be applied to a representation of uncertainty. Some of this is joint work is with Nir Friedman.
Joseph Y. Halpern received a Bs.C. in mathematics from the University of Toronto in 1975 and a Ph.D. in mathematics from Harvard in 1981. In between, he spent two years as the head of the Mathematics Department at Bawku Secondary School, in Ghana. After a year as a visiting scientist at MIT, he joined the IBM Almaden Research Center in 1982, where he remained until 1996; he was also a consulting professor at Stanford during that time. He then moved to Cornell University, where he is a professor in Computer Science and co-director of the Cognitive Studies program.
His major research interests are in reasoning about knowledge and uncertainty, qualitative reasoning, belief revision, (fault-tolerant) distributed computation, game theory, decision theory, and security. Together with his former student, Yoram Moses, he pioneered the approach of applying reasoning about knowledge to analyzing distributed protocols and multi-agent systems. He has coauthored 5 patents, a book (“Reasoning about Knowledge”), and well over 100 technical publications.
Halpern was program chairman and organizer of the first conference on Theoretical Aspects of Reasoning about Knowledge, and program chairman of the fifth ACM Symposium on Principles of Distributed Computing, the23rd ACM Symposium on Theory of Computing, and the16th IEEE Symposium on Logic in Computer Science. He received the Publishers’ Prize for Best Paper at the International Joint Conference on Artificial Intelligence in1985 (joint with Ronald Fagin) and in 1989, the 1997 Godel Prize (joint with Yoram Moses), and two IBM Outstanding Innovation Awards. He is a Fellow of the American Association for Artificial Intelligence. He is editor-in-chief of Journal of the ACM, and also serves on the editorial board of Information and Computation, Journal of Logic and Computation, Chicago Journal of Theoretical Computer Science, and Artificial Intelligence.
The research and development of teams of intelligent software agents and robots have fascinated RoboCup researchers for the last five years. We have actively researched on the integration of reasoning, perception, and action in teams of agents that need to face adversarial environments. Robotic soccer offered a pioneering concrete task for this research, both for software agents and robots. RoboCup today involves several new directions, including simulation and robot rescue tasks and humanoid robots. In this talk, Veloso will go in detail over the research challenges underlying teams of distributed software agents, small robots with off-board vision and computer control allowed, and fully autonomous robots and Sony legged robots. We have witnessed RoboCup significantly advancing the scientific state of the art of multiagent and multirobot systems. Veloso will introduce the main contributions, including robot design, multiagent learning, behavior architectures, perception, communication, localization, and opponent behavior modeling and recognition.
Manuela M. Veloso is Associate Professor of Computer Science at Carnegie Mellon University. She received her Ph.D. in Computer Science from Carnegie Mellon University in 1992, and a degree in Electrical Engineering in 1980, and an M.Sc. in Electrical and Computer Engineering in 1984 from the Instituto Superior Tecnico in Lisbon. Her long-term research goal aims at the investigation and effective development of teams of intelligent agents, in which cognition, perception, and action are combined to address planning, execution, and learning tasks, in particular in uncertain, dynamic, and adversarial environments. With her students, Prof. Veloso has developed teams of robotic soccer agents in three different RoboCup leagues, researching on fully autonomous simulation and robot agents, and in visual-servoed computer-controlled robots. She received the NSF Career Award in 1995 and the Allen Newell Medal for Excellence in Research in 1997. Professor Veloso is Vice-President of the International RoboCup Federation.
Verbmobil is a speaker-independent and bidirectional speech-to-speech translation system for spontaneous dialogs that can be accessed via GSM mobile phones. It handles dialogs in three business-oriented domains, with context-sensitive translation between four languages (English, German, Japanese, and Chinese). We show that in Verbmobil’s multi-blackboard and multi-engine architecture the results of concurrent processing threads can be combined in an incremental fashion. We argue that all results of concurrent processing modules must come with a confidence value, so that statistically trained selection modules can choose the most promising result. Packed representations together with formalisms for underspecification capture the uncertainties in each processing phase, so that the uncertainties can be reduced by linguistic, discourse and domain constraints as soon as they become applicable. One of the main lessons learned from the Verbmobil project is that the problem of speech-to-speech translation can only be cracked by the combined muscle of deep and shallow processing approaches.
Professor Wolfgang Wahlster is the Director of the German Research Center for Artificial Intelligence (DFKI). He has published more than 150 technical papers and five books on intelligent user interfaces. His current research includes multimodal interfaces, user modeling, speech translation, and life-like presentation agents. He has been the Scientific Director of the VERBMOBIL consortium on spontaneous speech translation (1993-2000), the largest AI project in Europe with more than 100 researchers. He has served as the Chair of IJCAI from 1991-1993, of ECCAI from 1996-2000, and as the President of ACL in 2000. Professor Wahlster is a Fellow of AAAI and ECCAI, a recipient of the Fritz Winter Award, and the European Information Technology Award. In 1998, he has been awarded the degree of Doctor Honoris Causa by Linköping University, Sweden. In 2000, he was the first AI researcher to receive the Beckurts Award, Germany’s most prestigious award for scientific innovations.