MUSE (Multidimensional User-Oriented Synthetic Environment): A New Approach to the Human-Computer Interface Using Precognitive Models of Perception
Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence. Videos. Pages 2069-2069.
© 1995 International Joint Conferences on Artificial Intelligence
Abstract
The human mind is capable of absorbing and processing large volumes of information. Most of this processing, however, occurs at a precognitive level, the results of which serve to alert the cognitive mind to areas of potential interest. The Multidimensional User-oriented Synthetic Environment, JluSE is an open-ended software shell that provides a new approach to interacting with computer-based information. By using a real-time, device-independent software design and incorporating both cognitive and experiential models of human perception, JluSE greatly enhances a person's ability to examine, interact with, and understand relationships in complex information space. A JluSE shell may be wrapped around data, models, simulations or even complete programs. Using a design, based on human functionality, it provides tools for the presentation, exploration, navigation, manipulation, and examination of information. Users experience a highly interactive environment, capable of dynamically mapping information into visual, auditory or kinesthetic representations.
Full text extracted automatically from the PDF; figures, tables and equations may differ. The PDF is the version of record.
1. Introduction
Richard Hamming said "The purpose of computing is insight, not numbers." Today, as the Information Age moves forward, computers touch all aspects of our lives. Far from achieving "insight", however, there is often a feeling that we are slowly sinking under the enormous volumes of data. Helping people to explore, question, and understand complex information is an important criterion for future computational environments. Highly interactive human-computer environments can allow practical solutions to some problems far more rapidly than either human or computer operating independently.
While a computer excels at performing routine types of computations, it is relatively inefficient at subjective problems such as complex decision making. The mind, on the other-band, is the most advanced device ever developed for information correlation, pattern recognition, and anomaly detection. All of our senses are superbly designed to collect, correlate, and analyze continuous streams of infonnation; to discard irrelevant information at high speeds; and to make in tuitive connections.
In 1991, Sandia National Laboratories began a project to develop a new approach for interacting with computer based information. As a result of this effort, a highly interactive, software shell, JluSE (Multidimensional User oriented Synthetic Environment) has been developed. This shell permits users to create immersive, experiential en vironments to facilitate the flexible and humanistic exploration multivariate information.
2. The Human Interface Model
User interaction with synthetic, immersive environments present an exciting and relatively uncharted area. Enhancing precognitive visual perception requires attention to such factors as bead-tracking, peripheral vision, shadow variation, and stereo vision. The auditory sense can be used both as an independent information channel and to enhance other channels. Data sanification can be used to provide auditory queues or baselines against which other occurrences can be compared (e.g., the ticking of a clock vs. the movement of data or objects). If utilized appropriately (and unobtrusively), auditory information representations can be monitored at a precognitive level, leaving the cognitive mind free to independently examine other information. One important concern, however, is achieving balance and not overloading any single sensory channel.
In concert with the idea of a humanistically organized software environment, five areas of human-computer interaction have been defined and investigated. They are Exploration, Navigation, Presentation, Interaction, and Examination. These five areas serve to define a functional arena for interactive environments and provide the foundation for device and model independent tools.
2.1 Exploration
This domain involves techniques for moving within and between various models and representations of information. To facilitate this modality, f.luSE creates a virtual craft. This craft can be characterized as a private, extended office that travels with the viewer through the environment. The visual representation of the craft provides the user with a personal frame-of-reference within the information space. The craft functionality (e.g., hovercraft, plane, tethercraft) provides the means to explore spatial, temporal, and other dimensions within the information space. J.luSE also recog nizes time as a potentially independent dimension of explo ration. In data where time is a parameter, the system per mits the user to control both the speed and direction of time flow. Another means of exploration is by teleportation.
Instantaneous teleportation between marked spatial, temporal and N-dimensionallocations is provided.
2.2 Navigation
These methods assist users in maintaining both location and reference (e.g., to keep from becoming lost). One such tool is the navigational map that is displayed on the side wall of the craft. This 'you-are-here' map automatically shows the position of every defined object in the space as well as the location and orientation of the craft. Since this map is maintained in real-time, it captures the location and motion of both object and craft positions. N-dimensional markers provide another navigational tool. Markers are stored as N-dimensional reference points. Users may use any portion of a marker's descriptor as a reference informa tion for teleportation. In addition, doorways or portals pro vide a way of moving between different information spaces.
2.3 Presentation
Presentation refers to ways of representing complex information. These include both visual and auditory rep resentations, as well as tactile, force-feedback and kinesthetic techniques. Within the virtual craft, users may invoke cockpit displays, which allow a virtual transparent bubble around the craft to be used for translucent displays of infor mation. Such displays are controlled dynamically and travel with the craft. Cockpit displays permit the user to expand the dimensionality of the space by adding additional, non obtrusive, information displays. Since they can be peripherally located in close virtual proximity to the user, the detail and/or perspectives can be controlled by simply moving the bead or body (e.g., leaning closer or turning).
2.4 Interaction
Interaction relates to command and control structures that permit users to communicate, and directly interact, with both the system and application program (e.g., to select op tions and interact with the environment). To facilitate paral lel human operations and minimize distracting elements, a humanistically efficient interactive environment should per mit the mapping of operations to different areas of the hu man senses. This permits sub-cognitive input to be maxi mized and sensory conflict to be minimized.
In 3-D environments reading can conflict with spatial concentration. Synthetic speech generation, however, can be used to transform most written or textual information into conversational speech. The auditory and visual channels op erate quite harmoniously in parallel, and listening to status or information doesn't normally create visual contlicts. Analogously, the act of requesting status or controlling state functions (e.g., turning things on and off) is often more effi ciently performed by voice command rather than by locating buttons (either physically or virtually).
2.5 Examination
This domain includes capabilities that permit the trans formation, correlation, and overlay of information. It typi cally is the most cognitive of the five functional areas. Examination frequently involves asking questions relating to an information base. One way of posing such questions is by means of the virtual, on-board computer.
Mathematical transformation is often used to examine and compress information, particularly when it varies over large dynamic ranges (e.g., semi-log, log or polar plots). This capability also exists in the jluSE environment, but techniques have been conceptually and mathematically ex tended to encompass more types of information and to in clude non-visual representations. For example, non-linear scaling can be used to require all objects of information are visible, regardless of size or distance. Similarly control of viewer and craft size permits either macroscopic overviews or microscopic examinations (note that such changes may require corresponding adjustments to the interpupilary distance). ·
3. Shared Environments
Permitting multiple people to both independently and simultaneously explore the same synthetic environment is a unique aspect of immersive technology. The shared memory architecture of JluSE is designed to enable and extend this capability. The architecture is based on a distributed, asynchronous communication model. The design supports heterogeneous operation at eacb site and minimizes the network synchronization and update traffic. The system not only supports multiple users, but permits each to maintain a personalized or private 'cockpit' within the shared space. The cockpits' information displays are individually configured and not accessible by others in the environment.
4. Conclusions
JluSE bas been developed and tested over the past three years and bas been very successfully applied to a wide vari ety of applications (e.g., CAD models, explosive welding simulations, medical data, seismic information, finite element analysis, circuit simulation, structural analysis, kinematic evaluation, topological data) and bas been wrapped around complete programs. The system currently supports flat screen, stereo, and VR operation (with full bead-track ing), voice recognition, sound synthesis, data sanification, and various commercial interactive devices. The fact that all JluSE applications run on any configuration or subset of these hardware devices (down to a monitor, keyboard and mouse) illustrates the hardware independence of the design.
The humanistic basis of the software design bas proven enormously effective, both in enabling people to interact with computer-based information and in speeding up the analysis and decision making process. One of the greatest surprises in applying JluSE to different applications bas been the frequency with which it permitted people to discover important aspects of their own applications of which they were previously unaware.
1 JluSE (pronounced 'muse') is trademarked; patents are pending.
2work performed at Sandia National Laboratories is supported by the U.S. Department of Energy under contract DE-AC04-94AL85000
@inproceedings{ijcai1995p276,
title = {MUSE (Multidimensional User-Oriented Synthetic Environment): A New Approach to the Human-Computer Interface Using Precognitive Models of Perception},
author = {Maples, Creve and Peterson, Craig},
booktitle = {Proceedings of the Fourteenth International Joint Conference on
Artificial Intelligence, {IJCAI-95}},
publisher = {International Joint Conferences on Artificial Intelligence},
editor = {Chris S. Mellish},
pages = {2069--2069},
year = {1995},
note = {Videos},
url = {https://ijcai.org/proceedings/1995/276},
}
TY - CPAPER TI - MUSE (Multidimensional User-Oriented Synthetic Environment): A New Approach to the Human-Computer Interface Using Precognitive Models of Perception AU - Maples, Creve AU - Peterson, Craig T2 - Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, IJCAI-95 PB - International Joint Conferences on Artificial Intelligence PY - 1995 SP - 2069 EP - 2069 UR - https://ijcai.org/proceedings/1995/276 ER -Copied
Machine-readable versions of this paper, for tools and AI agents: