Virtual World as Interface for Human-Robot Interaction

1 Introduction

GameBots [Adobbati et al., 2001] is a modification of the commercial 3D multi-player video game Unreal Tournament. We are creating an adaptation of GameBots for robotic collaboration and intelligent agent research called Virtual Synergy. Our system allows the robots to share two environments at once: the physical environment and a virtual environment hosted on a GameBots server, which can be accessed remotely via the Internet; we'll deal with that aspect in section 2. The server also allows a user to be present in the virtual environment and interact with the robot through various mechanisms, which will be discussed in section 3. Section 4 discusses the use of software agents. Finally, in section 5, we will address how the current system can be expanded to include intelligent agents to facilitate human-robot interaction.

2 Inhabiting Multiple Worlds

We are developing a robot team to participate in the AAAI Urban Search and Rescue (USAR) competition. One of the tasks for the robots is to explore a simulated disaster area and map the environment, locating walls, fallen objects, areas to avoid (for structural reasons), and victims.

The Virtual Synergy system provides a means for agents in the virtual world to create a three-dimensional map by placing objects that are detected in the real world in their corresponding locations in the virtual environment. Each robot on the team will exist in both the physical world and the virtual world of Virtual Synergy. The robot will receive information from its sensors in the real environment and the Virtual Synergy server. The server will act as a virtual sensor, but providing map information instead of data. Because multiple agents will process the information gathered from multiple sensors, we must deal with the issues of sensor fusion.

A similar system called Player/Stage has been developed for simulating sensors and robots in a virtual two­ dimensional world [Gerkey et al., 2001]. Both Player/Stage and GameBots are designed to allow heterogeneous systems to communicate with the server over an IP network. Player/Stage sends simulated sensor readings to the client and the client sends actuator commands back to Player/Stage. The Virtual Synergy system, however, takes a higher-level approach; it provides mapping and path information; it also has the advantage of a three-dimensional view and the ability for humans to take on a persona in the environment.

3 Human Interactions

The nature of disaster areas often prohibits humans from being physically present; therefore, it is necessary to monitor and direct the robots from a distance, occluded by walls and debris. Research in teleoperation has focused on multiple operators controlling one robot. Murphy et al. suggest that teleoperation can lead to "cognitive fatigue" of the operators during long periods of operation and that the information received by the user often exceeds the user's capacity to process it [Murphy, 2000] . Our system attempts to resolve these issues with an intuitive interface that mimics physical interaction with the robots. Research dealing with multiple users interacting with multiple robots has shown the need for "adjustable autonomy," in which a robot can be controlled to varying degrees by user intervention [Kortenkamp et al., 2002]. Multiple levels of adjustable autonomy will be possible with the Virtual Synergy system.

A user in the environment of a Virtual Synergy server has various modes of navigation; he/she can fly unimpeded by gravity, walk through walls, or use a more realistic physics model. The interface for navigation is intuitive; it was chosen because it was designed for navigation of a three-dimensional environment by multiple human players.

We are researching the use of user roles to facilitate user interactions with robots. Current research [Scholtz, 2002] defines three levels of interaction with a robot: supervisor, peer, and mechanic interactions. The different roles define the information displayed to the user and the set of possible interactions available to the user. In addition to the three roles described in [Scholtz, 2002], we add two other roles: spectators can view the environment but not interact with the robots and teleoperators take complete control of the robot by temporarily inhabiting its representation in order to direct it through areas of difficulty.

Perzanowski et al. have developed a system for controlling teams of robots using voice and gesture commands [Perzanowski et al., 2002]. Our robotic interactions are currently not based on speech or gestures, but on direct manipulation of the robot's world model. While navigating in the virtual environment, the user will instruct the robots through various commands such as "come here." The commands are English-like only for purposes of user interface (natural language and gesture commands are a foreseen addition). The user can also directly modify the environment by leaving a path to follow or indicating "hot spots" which suggest areas to explore. The system also provides the user the ability to manipulate the environment in order to rectify any sensor anomalies.

4 Software Agents

Autonomous software agents will also inhabit the virtual environment as teammates. Agents can facilitate communication between the users and the robots, effectively providing a more natural, more human interface to dealing with information from the robots. Since all agents (robots, software agents, and people) in the virtual environment are collaborators on the same team, they can all take on the different roles with each other; in addition to carrying out user-specified tasks, the software agents could also autonomously lead a group of robots with minimal human supervision.

5 Conclusion and Future Work

The ultimate goal of our project is to facilitate human interaction with a team of robots and agents. We are researching ways to increase the efficiency of robot collaborators; our aim is to decrease the number of humans required to monitor and collaborate with a larger team of robots without overloading the users with information from an increasingly complex team of autonomous agents.

The two problems we are immediately facing are sensor fusion and localization. Localization is important for creating an accurate map of the environment; sensor fusion becomes a problem once two robots are involved in building the map: they could potentially be looking at the same object. We would not want the same object to appear twice.

The use of two environments, one virtual, one actual , allows for complex human-robot interaction and useful inter-robot collaboration. Using the 3D environment of Virtual Synergy will give the user an intuitive interface to interact with the robots as another member of the team. User roles will act to define the possible interactions with the robots. Intelligent agents will assist the user by responding to queries, gathering information, and automating tasks.