On Teaching Collaboration to a Team of Autonomous Agents via Imitation

1 Introduction

Though coordination and teamwork are rudimentary parts of multi-agent systems, training agents to work in coordination and exhibit teamwork is a tedious task. In complex multiagent domains, acquiring knowledge from human experts and effectively transferring it to multiple agents requires powerful yet intuitive mechanisms of knowledge transfer. Most of the existing approaches ask human experts to articulate their process of decision-making in the form of rules, state-machines, or cost/utility functions. Though humans have inherent ability to make intelligent decisions during complex situations, structuring the knowledge in a manner that can be used by autonomous agents is a nontrivial task that hampers the overall process of knowledge transfer. In order to fill this gap and make this knowledge transfer mechanism natural and intuitive, imitation learning has emerged as a popular choice in recent years. This study proposes the use of imitation based learning to formulate collaborative strategies and demonstrates its effectiveness in the domain of RoboCup Soccer Simulation 3D. We have already established a RoboCup Soccer Simulation 3D team that participated in World RoboCup 2011 and 2012 and this team would serve as an evaluation benchmark for this research.

2 Related Work

Various studies have been conducted, primarily in the field of robotics, which propose the idea of learning via demonstration to train robots. Some supervised learning based imitation include controlling a car within a simulated driving domain using Gaussian Mixture Models (GMMs) (Chernova and Veloso, 2009), flying a simulated airplane using decision trees (Sammut et al., 1992) and learning obstacle avoidance and navigation behaviors using kNN classifiers (Saunders et al., 2006). Argall et al. (Argall et al. 2009) presents a survey of different imitation learning approaches. RoboCup Soccer has been a popular test bed for strategy building and optimization. Aler et al. (Aler et al. 2009) have applied imitation based learning and evolutionary algorithms to build and optimize RoboCup Soccer strategies while Wai (Wai, 2011) proposed the use of Inductive Logic Programs to train RoboCup Soccer agents via imitation.

Most of the imitation based learning approaches proposed in the literature are focused on the problem of training a specific task to an individual agent. Very few studies have talked about teaching a team of robots how to perform a complex task in coordination with each other. Chernova and Veloso (Chernova and Veloso, 2009) have addressed the problem of teaching a joint task to multiple agents via imitation and performed experiments in a simulated furniture mover domain to teach two agents how to move a couch from one place to another.

3 Proposed Approach

The proposed approach of imitation learning, as shown in Figure 1, is divided into two phases: the demonstration phase and the learning phase. The demonstration phase will be performed by multiple human experts, each expert controlling a different agent, and all of them collectively demonstrating a collaborative strategy. The state-action data gathered during this phase will then be fed to the learning phase where a supervised learning mechanism will be applied to learn the strategy exhibited by the experts. Such multi-human demonstrations immediately raise the concerns of limiting a) the number of demonstrators involved and b) the amount of demonstration.

To limit the number of demonstrators, this research introduces the notions of roles and sub-teams. The team will be divided into sub-teams that comprise of certain predefined roles. The execution of a role and coordination among multiple roles within sub-teams will be learned via imitation. When all the sub-teams are trained, they will be glued together using a mechanism of dynamic role assignment. The layer of dynamic role assignment will sit on the top of strategy execution module and based on the situation of the game, will assign players to different teams. Once assigned to a team, the player will follow imitation based model to exhibit a collaborative strategy.

To reduce the overhead of demonstration, an online learning mechanism will be used to incrementally refine the already learned model. This refinement will ensure that more demonstration efforts are required in the beginning when agents are completely in an ignorant state than in the later phases of demonstration when agents have already acquired sufficient knowledge and only ask for human assistance/feedbacks, as or when required, to tackle unfamiliar situations.

4 Experiments

So far, we have experimented with some preliminary experiments for soccer agents to learn different tasks via imitation (Raza and Haider 2012; Raza et al. 2012). These tasks include:

• ♥ A single player scoring goals in an empty soccer field
• ♥ A single player avoiding collision with obstacles
• ♥ Two players defending the goal

We gathered data by demonstration and built classification model on this data to imitate human behavior. The results have been very encouraging and the performance of our imitating agent is comparable to human-driven agents.

Acknowledgments

The work was partially supported by Higher Education Commission, Pakistan research grant 20-1815/RCD/105032.