Augmented Experiment: Participatory Design with Multiagent Simulation

1 Introduction

Ubiquitous/pervasive computing systems in the public space often interact with anonymous users. We propose to apply multiagent technologies to the design of those systems that are to be embedded in society.

Participatory simulations have been already studied intensively for modeling human societies [Drogoul et al., 2002, Guyot et al., 2005]. For designing socially embedded systems, however, simulations in virtual space cannot reproduce the reality of the actual application environment. The goal of this paper is to introduce real world experiments to the process of participatory design, so as to bridge the gap between participatory simulations and services in operation.

ologies (participatory design hereafter) to test socially embedded systems in the following steps.

1. Describe interactions between human users and socially embedded systems so as to define the expected user behaviors with socially embedded systems (interaction model hereafter).
2. Perform a multiagent-based simulation by modeling users under the given interaction scenarios (agent model hereafter). The simulation takes place in virtual space and involves a large number of simulated users. Results of the simulation can predict how the entire system would work in society.
3. Replace some of the simulated users by human-controlled avatars to perform a multiagent-based participatory simulation (participatory simulation hereafter). The simulation is performed in virtual space, and the avatars are controlled by human subjects sitting in front of their personal computers.
4. Perform experiments in real space to try out the entire system with human subjects. Since the number of subjects is often limited, the experiment should be augmented by a large number of simulated users in virtual space. We called this the multiagent-based augmented experiment (augmented experiment hereafter).

To learn from the participatory design process, this paper separates agent models from interaction models: the former covers the beliefs, desires, intentions, and emotions of human users, and the latter covers protocols, methods, rules, or laws that guide users when interacting with the socially embedded systems. We use extended finite state automata for describing interaction models, while production rules are used to describe agent models for approximating users. We view interaction models as behavioral guidelines of human users playing with socially embedded systems; users keep autonomy within the given guidelines. The question is whether or not users will follow the guidelines in an actual service environment. In other words, the effectiveness of interaction models depends on the agent models, which include user personalities such as deliberative and reactive.

In this paper, we call the descriptions of interaction models scenarios. We use scenario description language Q, which can simultaneously interpret scenarios for a large number of agents [Ishida, 2002b]. Q has been already connected to FreeWalk [Nakanishi et al., 2004b] for virtual city experiments [Ishida, 2002a], Cormas for a natural resource simulation [Bousquet et al., 1998], and Caribbean [Yamamoto et al., 1999] for massively multiagent simulations [Ishida et al., 2005]. Q, FreeWalk, and Caribbean have been used to conduct indoor and outdoor disaster evacuation experiments. Those real-scale experiments confirm the feasibility and usefulness of augmented experiments.

2 Participatory Simulation

In our participatory design methodology, we first conduct multiagent-based simulations. Figure 1(a) illustrates how a multiagent-based simulation is realized. The scenario processor interprets interaction models and requests agents in virtual space to perform sensing and acting functions. Note that, since agents are autonomous and have their own agent models, though agents receive instruction according to the scenarios, there is no guarantee that they will behave as requested.

We can easily extend multiagent-based simulations to yield participatory simulations by replacing some of the scenario-guided agents with human-controlled avatars. Figure 1(b) illustrates how human subjects and agents can cooperatively perform a simulation. Just as with video games, human subjects can join the simulation by controlling avatars via joy sticks, mice, or other input devices. To analyze simulation results, we monitor the entire process of the simulation by visualizing the virtual space. In addition to video taping the virtual space, we record how the human subjects control their avatars. Recording human behavior is useful for analyzing the simulation results and for improving the agent and interaction models.

In summary, a participatory simulation consists of 1) agents for modeling users, 2) avatars to represent human subjects, 3) scenarios for modeling interactions, 4) human subjects to control avatars, 5) virtual space to represent real space, and 6) a monitor to visualize simulations ongoing in the virtual space.

We conducted a multiagent-based simulation in the disaster evacuation domain. To establish the process of refining agent models and interaction models, we simulated the precisely controlled experiments conducted by Sugiman [Sugiman et al., 1998]. He developed a simple environment with human subjects to determine the effectiveness of two evacuation methods (i.e., interaction models): the Follow-direction method and the Follow-me method. In the former, the leader shouts out evacuation instructions and eventually moves toward the exit. In the latter, the leader tells a few of the nearest evacuees to follow him and actually proceeds to the exit without verbalizing the direction of the exit.

A participatory simulation was performed in which we replaced twelve out of twenty scenario-guided agents by human-controlled avatars. The virtual space was designed so that the human subjects could not distinguish agents from avatars. After the simulation, we applied deductive machine learning technology with domain knowledge to refine the agent models from the logs of the participatory simulation, and successfully modified the agent models and original evacuation scenarios (i.e., interaction models).

Once an accurate model is acquired from participatory simulations, it becomes possible to simulate experiments that cannot be conducted in real space. For example, Sugiman's experiment was performed using two or four leaders, but we can vary this number to clarify the relation between the number of leaders and the time required for evacuation. Moreover, we can explore an effective combination of the Follow-me and Follow-direction methods. Given accurate and explainable agent models, even though the models can only capture a part of the variations possible in human behavior, participatory simulations must be seen as useful in educating or training people. People can be provided with a learning environment wherein evacuations or other drills can be experienced.

3 Augmented Experiment

Participatory simulations are particularly useful when conducting controlled experiments: they make it easy to prepare the application environments for testing, and user behavior can be recorded for later analysis. However, it is sometimes fails to provide enough reality to allow the testing of ubiquitous/pervasive computing environments.

To understand how users behave with socially embedded systems, real-world experiments are often required. In the case of video phones, for example, since user behavior in everyday life is not easy to simulate, it is essential to observe how individual users employ their video phones in real space. In ubiquitous/pervasive computing, however, because a large number of electronic devices will be embedded in public spaces like railway stations and are in continuous service, it is desired but often impossible to conduct experiments with a large number of human subjects. Augmented experiments offer the ability to use just a small number of human subjects to perform an experiment that can be enhanced by large scale multiagent-based simulations.

Figure 2 illustrates how augmented experiments are realized. Figure 2(a) represents a real world experiment, where human subjects communicate and perform an experiment in real space. Figure 2(b) shows how we introduce a virtual space into the real world experiment. The sensors in the real space captured the behavior of human subjects for reproduction in the virtual space. The sensors can be cameras, RFIDs, or GPS depending on the environment. By using the virtual space, we can monitor the entire experiment from various viewpoints. Furthermore, we can communicate with the human subjects in the real space through their avatars in the virtual space. Transcendent communication is a new monitoring interface, where a visually simulated public space provides a more flexible and consistent view than regular surveillance systems [Nakanishi et al., 2004a].

Figure 2(c) illustrates an augmented experiment. In parallel with a real world experiment, a large scale multiagent-based simulation is conducted in virtual space: the experiment is augmented by the simulation. To provide social reality to the human subjects, scenario-guided extras are placed around the subjects. In contrast to the participatory simulations, the human extras do not control avatars: the human extras in the real space are controlled by the scenario-guided agents. For example, we can use human extras acting as evacuation leaders to conduct disaster evacuation drills. In contrast to participatory simulations where human subjects sense and act in virtual space, augmented experiments allow subjects to sense and act in real space. Conducting an augmented experiment is possible if the real space is equipped with enough sensors.

1. Real-World Experiment a) Real-World Experiment
2. Virtual Space to Monitor Experiment b) Virtual Space to Monitor Experiment

Figure 2. Augmented Experiment

In summary, an augmented experiment consists of 1) agents for modeling users, 2) avatars to represent human subjects, 3) scenarios for modeling interactions, 4) human subjects to act experiments, 5) virtual space to represent real space, 6) a monitor to visualize the experiment in real space enhanced by simulations in virtual space, 7) sensors to reproduce human subjects in virtual space as avatars, 8) communication channels between real and virtual spaces, and 9) human extras to represent agents in virtual space for interacting with human subjects.

The issues on augmented experiments are as follows.

1. Seamless connections between virtual and real spaces: The difficulty with augmented experiments is to provide the human subjects with a sufficient level of reality. To guide human subjects in a timely fashion, the subjects
2. must receive sensing information through the communication channels or from the corresponding human extras. To ensure the subjects naturally behave in real space, the selection of sensor devices and communication media becomes crucial. Visual, auditory and textual interactions are to be organized appropriately.
3. Learning from data obtained in virtual and real spaces: Learning issues are similar to those in participatory simulations, but their evaluation is more difficult, since the behaviors of human subjects in real space are harder to analyze than those in participatory simulations. Though we can monitor the experiment via virtual space, video capture of the human subjects in real space is also indispensable for analyzing the entire experiment.

To confirm the feasibility and usefulness of augmented experiments and to determine their future issues, we conducted indoor and outdoor evacuation experiments in real space augmented by a large scale multiagent-based simulation.

4 Indoor Experiment

Everyday, more than 300,000 passengers pass through Kyoto station, the main railway station in Kyoto. In this station, using Q [Ishida, 2002b] and FreeWalk [Nakanishi et al., 2004b], we installed a disaster evacuation system that tracks passengers to help their navigation based on their current positions. Beyond conventional navigation systems, which announce route information using public loudspeakers, our system sends instructions to individuals using mobile phones. Augmented experiments are required for testing the evacuation system, because there is no other way to conduct experiments with enough reality.

3D Virtual Space

3D Virtual Space

Indoor Real Space Indoor Real Space

Figure 3. Indoor Experiment

The augmented experiment was designed as follows. As the sensors, we placed twenty eight cameras in Kyoto station, and captured the movements of passengers in real time. The cameras are specially designed to detect passengers' behavior but not personal features. Figure 3 shows the omnidirectional cameras placed in the station. As the virtual space, we used FreeWalk, a three dimensional virtual city system and used it to reproduce the passengers' behavior. We implemented a monitor based on transcendent communication [Nakanishi et al., 2004a]. Figure 3 includes a snapshot of the monitoring system; human subjects on the station platform are projected onto avatars in virtual space. A bird's-eye view of the real space is reproduced on the screen of the control center so that evacuation leaders in the center can easily monitor the experiment. As the communication channels, the leader can point at particular passengers on the screen, and talk to them through their mobile phones. When the monitor detects pointing operations, a wireless connection is immediately activated between the control center and the indicated passengers.

A multiagent-based simulation with a large number of agents controlled by evacuation scenarios was performed in parallel with the experiment in real space. See-through head-mounted displays are not suitable for presenting the simulation of augmented experiments, since it is unsafe to mask the views of passengers. As described above, since we used mobile phones, small and low-resolution images of three dimensional virtual spaces are difficult to understand. Instead of displaying visual simulations, the mobile phones displayed symbols that directly represent what human subjects need to understand. We did not use human extras in this experiment.

We conducted the indoor augmented experiment in February 2005. From this experience, we learned that insufficient visual reality prevented the human subjects from recognizing the crowd of agents around the staircase. It appears that the usability of an augmented experiment depends significantly on the user interface for interacting with the agents.

5 Outdoor Experiment

One year after the indoor experiment, we implemented a large-scale outdoor evacuation system for emergency situations using Q [Ishida, 2002b] and Caribbean [Yamamoto et al., 1999], and conducted an augmented experiment in January 2006. This system architecture is close to that of our indoor experiment, but the sensors were GPS devices instead of omnidirectional cameras, and as the virtual space, we used a two dimensional map instead of a three dimensional virtual city system. The virtual space is displayed on the monitor screen of the control center in a birds-eye view, so that the leader could grasp how evacuees were behaving in the experiment. Evacuees on a screen consisted of a small number of avatars representing the human subjects in real space, and a large number of agents controlled by evacuation scenarios in virtual space. In the actual experiment, ten to thirteen humans and three thousand agents per each time undertook the augmented experiment.

The location of human subjects in real space was projected into virtual space based on their positions acquired by GPS. This map showed fires, blocks routes, and safe areas in real time. The human subjects could always get the latest map by sending their location. The evacuation leaders assigned evacuation destinations and directions to each evacuee through the monitor screen shown in Figure 4. The leader issued high level instructions to the evacuees using the map, and precise navigation instructions were automatically generated for each evacuee. Dragging operations indicating a rectangular area enabled the leader to broadcast an announcement to a group of evacuees.

As in the indoor experiments, the evacuation leader activated communication channels to particular evacuees by pointing at the evacuees on the screen. The leader could talk to the particular evacuees and provide customized navigation. We used human extras as evacuation leaders in this experiment to check the evacuation routes taken by human subjects in real space.

The human subjects could grasp the experiment from the screen of their mobile phones. The map around the place the subject was standing was displayed together with information of fires, blocked routes, and safe areas, just as on the monitor in the control center. The moves of other evacuees were also displayed on their mobile phones. From this experience, we learned that a map can be an excellent interface between human subjects and agents as well as evacuation leaders. Unlike the indoor experiment, since route selection is the main issue in the outdoor experiment, the human subjects did not have much difficulty in imagining the disaster situation. However, if the number of agents was high in the outdoor experiment, then virtual crowding would become a problem. Human extras playing evacuation leaders can increase the reality of the experiment.

6 Learning from Experiments

Multiagent-based simulations use production rules to describe agent models for approximating human users, and extended finite state automata to describe interaction models among users and socially embedded systems. By analyzing the logs of participatory simulations or augmented experiments, we can refine both agent and interaction models. The learning process of multiagent-based participatory design consists of two phases: the normal design loop and the participatory design loop.

At first, scenario writers create agent and interaction models and verify their design by multiagent-based simulations. This process is called the normal design loop and its detailed steps can be found in a previous work [Murakami et al., 2003]. After verifying the agent and interaction models, participatory simulations and augmented experiments are performed. The results of which may be different from those of multiagent-based simulations, because human behaviors are often more variable than those of agents modeled by scenario writers. If this is the case, we first refine the agent models to better express the variation seen in human behavior. We then perform the multiagent-based simulations again with the new agent models. If the new simulations succeed to express various human behaviors but fail to produce the expected results, we then refine the interaction models; i.e. modify protocols, methods, rules, and/or laws. After the improved interaction model is confirmed by multiagent-based simulations, participatory simulation and augmented experiment take place again. This loop, called the participatory design loop, is illustrated in Figure 5.

It would seem, at first glance, that inductive machine learning technologies could be applied to refine agent and interaction models. However, it is extremely rare to have a sufficient number of human subjects to gather enough data for inductive learning: deductive methods with domain knowledge are more feasible [Torii et al., 2006]. To refine agent models, given that the training examples will be limited, explanation-based learning is a reasonable approach. Though variations exist in human behavioral rules such as "escape from crowds" and "follow crowds," observing the behavior of human-controlled avatars can create a plausible explanation that selects one of the two rules.

By combining machine learning technologies with interviews of human subjects, agent models have been successfully refined in several domains [Murakami et al., 2005]. To refine interaction models, if we have enough candidate protocols, we can try them, and select the one that best fits. If we need to invent a new model, however, there is no available technology that can create such an interaction model. The visualization of simulations and experiments can, however, support scenario writers in inventing new models.

7 Related Work

There are two types of multiagent-based simulations and they have different goals; a) analytic multiagent-based simulations often with simple agent and interaction models and b) synthetic multiagent-based simulations sometimes with complex agent and interaction models. Analytic simulations have been used to model social systems [Epstein and Axtel, 1996]. Here, the KISS principle (Keep It Simple, Stupid) is often applied [Axelrod, 1997]. The KISS principle states that modeling should be simple even though the observed phenomenon is complex, and that complexity should be a result of repeated interaction among agents and their environment. Hence, agents are often modeled with limited parameters. This approach is effective in the analysis of macro-micro links, the relationship between the macro properties of the entire system and the micro properties of the agents constituting the system.

On the other hand, synthetic simulations are used to produce realistic situations. Agent and interaction models can be complex reflecting the real world to make the simulation as realistic as possible. This approach is used in an early stage of system development [Noda and Stone, 2003], and in education or training [Rickel and Johnson, 2000]. Since our purpose is to design socially embedded systems, our simulation is synthetic: our agent and interaction models can be complex to reproduce realistic situations; after a participatory simulation, an augmented experiment is performed to understand user behavior in a more realistic environment.

Our work appears to follow various multiagent-based simulation papers including the RoboCup soccer simulator [Noda and Stone, 2003], but is clearly unique due to its focus on participatory design methodology. Drogoul proposed a methodological process for developing multiagent-based simulations, and introduced the idea of the participatory design of simulations [Drogoul et al., 2002], while we are proposing the participatory design for socially embedded systems. Bousquet applied role games to the modeling of social systems [Bousquet et al., 2002]. This paper, however, applies multiagent-based simulation to software system design, not social system analysis.

We propose the use of participatory methodology including simulations and experiments for designing large-scale socially embedded systems. This methodology differs from the Gaia methodology [Zambonelli et al., 2003] or OperA [Vázquez-Salceda et al., 2005], which focus on an early stage of designing multiagent systems mainly for enterprise applications [Jennings, 2000]. Our methodology can be applied to designing ubiquitous/pervasive computing systems, but the resulting systems are not necessarily multiagent-based, though multiagent systems are often natural solutions of socially embedded systems.

8 Conclusion

The waterfall model has been used in software development for a long time. Given the increase in human-computer interaction, however, it has become essential to use the user-centered design approach when creating usable and accessible interactive systems. In future, however, millions of electronic devices with computing facilities will be connected with each other in ad hoc ways, and should behave coherently. It is natural to say we need a new methodology for designing ubiquitous/pervasive computing systems.

In this paper, we proposed the multiagent-based participatory design methodology for ubiquitous/pervasive computing systems, which are to be embedded in human society. Though the concept of multiagent-based participatory design is open-ended, we include the following two components.

We call multiagent-based simulations participatory simulations when they include human-controlled avatars and scenario-guided agents. We can monitor the entire process of a participatory simulation by visualizing the virtual space. We call real-world experiments as augmented experiments when they are augmented by large scale multiagent-based simulations. In ubiquitous/pervasive computing, because a large number of electronic devices will be embedded in public spaces and in continuous use, it is often impossible to conduct experiments with a large number of human subjects. Augmented experiments enable us to perform such trials with a small number of human subjects.

By introducing augmented experiments and actually trying out indoor and outdoor experiments in the city of Kyoto, we learned the following lessons. Interaction among the human subjects in the real space and agents in the virtual space should be designed carefully. If there exists a medium like geographical map, which human subjects use daily to grasp situations, it is a good idea to adopt them as the interface between the real and virtual spaces. Human extras controlled by scenario-guided agents are also effective to increasing the reality, if they have a clear role such as evacuation leaders.