Autonomous Cross-Domain Knowledge Transfer in Lifelong Policy Gradient Reinforcement Learning

1 Introduction

Reinforcement learning (RL) provides the ability to solve high-dimensional control problems when detailed knowledge of the system is not available a priori. Applications with these characteristics are ubiquitous in a variety of domains, from robotic control [Busoniu et al., 2008; Smart & Kaelbling, 2002] to stock trading [Dempster & Leemans, 2006]. However, in many cases, RL methods require numerous lengthy interactions with the dynamical system in order to learn an acceptable controller. Unfortunately, the cost of acquiring these interactions is often prohibitively expensive (in terms of time, expense, physical wear on the robot, etc.).

This issue of obtaining adequate experience only worsens when multiple control problems must be solved. This need arises in two main cases: (1) when a single agent must learn to solve multiple tasks (e.g., a reconfigurable robot), and

Jos´e Marcio Luna

Univ. of Pennsylvania joseluna@seas.upenn.edu

Paul Ruvolo

Olin College of Engineering paul.ruvolo@olin.edu

(2) when different robots must each learn to solve a single control problem. Transfer learning [Taylor & Stone, 2009] and multi-task learning (MTL) methods [Li et al., 2009; Lazaric & Ghavamzadeh, 2010] have been developed to reduce the amount of experience needed for individual tasks by allowing the agent to reuse knowledge from other tasks.

In both transfer learning and MTL, the difficulty of transferring knowledge between tasks increases with the diversity of the tasks. In the extreme case, the underlying systems (and their action and state representations) are entirely different, making direct knowledge reuse impossible. Several approaches to this cross-domain transfer problem have been developed, but these methods require an inter-task mapping of state/action spaces that is either hand-coded [Taylor et al., 2007] or learned in a computationally inefficient manner that does not scale to more than a few task domains (see Section 3). However, the problem of cross-domain transfer has not yet been studied in lifelong learning settings [Thrun & O'Sullivan, 1996; Ruvolo & Eaton, 2013], in which the agent must learn multiple tasks consecutively with the goal of optimizing performance across all previously learned tasks.

We address this problem by developing the first algorithm for lifelong RL that supports efficient and autonomous crossdomain transfer between multiple consecutive tasks from different domains. Specifically, our approach provides the following advantages over existing approaches: (1) it improves current cross-domain transfer learning methods by optimizing performance across all tasks, (2) it learns multi-task crossdomain mappings autonomously, and (3) it can share knowledge between a multitude of tasks from diverse domains in a computationally efficient manner. To enable effective crossdomain lifelong learning, our approach learns a repository of shared knowledge along with projection matrices that specialize this shared knowledge to each task domain. We provide theoretical guarantees on convergence that show this approach becomes increasingly stable as the number of domains or tasks grows large, and demonstrate the empirical effectiveness of cross-domain lifelong learning between streams of interleaved tasks from diverse dynamical systems, including bicycles and helicopters.

2 Background on Policy Gradient RL

In a reinforcement learning (RL) problem, an agent must decide how to sequentially select actions to maximize its expected return. In contrast to classic stochastic optimal control methods [Bertsekas, 1995], RL approaches do not require detailed prior knowledge of the system dynamics or goal; instead these approaches learn optimal control policies through interaction with the system itself. RL problems are typically formalized as a Markov decision process (MDP) 〈X,A, P,R, γ〉, where X ⊆ Rd is the (potentially infinite) set of states, A is the set of actions that the agent may execute, P : X × A × X → [0, 1] is a state transition probability function describing the task dynamics, R : X × A × X → R is the reward function measuring the performance of the agent, and γ ∈ [0, 1) is the reward discount factor. At each time step m, the agent is in state xm ∈ X and must choose an action am ∈ A, transitioning it to a new state xm+1 ∼ P (xm+1 | xm, am) and yielding a reward rm+1 = R(xm, am, xm+1). The sequence of state-action pairs forms a trajectory τ = [x1:M, a1:M] over a (possibly infinite) horizon M. A policy π : X × A → [0, 1] specifies a conditional probability distribution over actions given the current state. The RL agent's goal is to find a policy π⋆ that maximizes the expected per-time-step reward.

Policy gradient methods [Peters et al., 2005; Sutton et al., 1999] represent the agent's policy π as a function defined over a vector θ ∈ Rd of control parameters. With this parameterized policy, we can compute the optimal parameters θ⋆ that maximize the expected average reward:

where T is the set of all possible trajectories, R(τ) = 1M ∑Mm=1 rm+1 is the reward of trajectory τ, and pθ(τ) = P0(x1) ∏Mm=1 P(xm+1 | xm, am)πθ(am | xm) is the probability of τ with initial state distribution P0 : X → [0, 1].

To maximize J(·), most policy gradient algorithms (e.g., episodic REINFORCE [Williams, 1992], PoWER [R¨uckstieß et al., 2008], and Natural Actor Critic [Peters & Schaal, 2008]) employ standard supervised function approximation to learn θ by maximizing a lower bound on J (θ). To maximize this lower bound, these methods generate trajectories using the current policy πθ, and then compare the result with a new policy π˜ θ . Kober & Peters [2011] describe how this lower bound on the expected return can be attained using Jensen's inequality and the concavity of the logarithm:

where DKL (p(τ) || q(τ)) = ∫ Tp(τ) log p(τ) q(τ) dτ. Consequently, this process is equivalent to minimizing the KL divergence DKL between πθ's reward-weighted trajectory distribution and the trajectory distribution p˜ θ of π˜ θ .

Policy gradient methods have gained attention in the RL community in part due to their successful applications to robotics [Peters et al., 2005]. While such methods have a low computational cost per update, high-dimensional problems require many updates (by acquiring new rollouts) to achieve good performance. Transfer learning and multi-task learning can reduce these data requirements and accelerate learning.

3 RelatedWork on RLKnowledge Transfer

Knowledge transfer between tasks has been explored in the context of transfer learning and multi-task learning. In all cases, each task t is described by an MDP Z (t) = 〈X (t) , A(t) , P (t) , R(t) , γ (t) 〉.

Transfer learning aims to improve the learning time and/or behavior of the agent on a new target task by transferring knowledge learned from one or more previous source tasks [Taylor & Stone, 2009]. However, transfer learning methods only focus on optimizing performance on the target task, and therefore are not ideal for agents that revisit earlier tasks. In contrast, multi-task learning (MTL) methods [Li et al., 2009; Lazaric & Ghavamzadeh, 2010] optimize performance over all tasks, often by training task models simultaneously while sharing knowledge between tasks, but are computationally expensive in lifelong learning scenarios where the agent must learn tasks consecutively over time [Ruvolo & Eaton, 2013; Thrun & O'Sullivan, 1996], as we explore in this paper.

One exception is PG-ELLA [Bou Ammar et al., 2014], a recent lifelong policy gradient RL algorithm that can efficiently learn multiple tasks consecutively while sharing knowledge between task policies to accelerate learning. In fact, PG-ELLA can be viewed as a special case of the algorithm we propose in this paper where learning is limited to only tasks from a single domain. MTL for policy gradients has also been explored by Deisenroth et al. [2014] through customizing a single parameterized controller to individual tasks that differ only in the reward function. Another closely related work is on hierarchical Bayesian MTL [Wilson et al., 2007], which can learn RL tasks consecutively, but unlike our approach, requires discrete states and actions. Snel and Whiteson's [2014] representation learning approach is also related, but assumes all tasks share the same feature and action sets. All of these MTL methods operate on tasks from a single domain, and do not support cross-domain transfer.

To enable transfer between tasks with different state and/or action spaces, transfer learning and MTL methods require an inter-task mapping to translate knowledge between tasks. The earliest work on cross-domain transfer in RL, by Taylor et al., required a hand-coded mapping [2007] or a computationally expensive exploration of all permitted mappings [2008]. More recently, unsupervised techniques have been developed to autonomously learn inter-task mappings [Bou Ammar et al. 2012; 2013]. While these approaches enable autonomous cross-domain transfer, they only learn pairwise mappings between tasks and are computationally expensive, making them inapplicable for transfer among numerous tasks from different domains. In contrast to these methods, our approach provides a computationally efficient mechanism for transfer between multiple task domains, enabling cross-domain lifelong RL. Cross-domain MTL has also been explored in a limited fashion in supervised settings [Han et al., 2012].

4 ProblemDefinition

Previous work on policy gradients has focused on either single-task learning or MTL within a single task domain (i.e., all tasks share a common state and action space, but may differ in other aspects). We focus on the problem of learning multiple tasks consecutively, where the tasks may be drawn from different task domains. Specifically, the agent must learn a series of RL tasks Z (1) , . . . ,Z (Tmax) over its lifetime, where each task Z (t) is an MDP and the tasks may have different state and/or action spaces. We can partition the series of RL tasks into task groups G(1) , . . . , G(Gmax) , such that all tasks within a particular group G(g) (i.e., a set of tasks) share a common state and action space, and are generated by varying latent parameters [Konidaris & Doshi-Velez, 2014]. In our experiments, each task group corresponds to a task domain-a class of dynamical systems, such as helicopters, cart-poles, etc., each of which contains multiple tasks corresponding to multiple physical helicopters or cart-poles. However, this framework can easily generalize so that a particular group G(g) could represent a subset of tasks from a domain, similar to task clustering frameworks [Kang et al., 2011].

The agent must learn the tasks consecutively, acquiring multiple trajectories within each task before moving to the next. The tasks may be interleaved, offering the agent the opportunity to revisit earlier tasks (or task domains) for further experience, but the agent has no control over the task order. We assume that a priori the agent does not know the total number of tasks Tmax, their distribution, the task order, or the total number of task groups Gmax. The agent also has no prior knowledge about the inter-task mappings between tasks, and so it must also learn how to transfer knowledge between task domains in order to optimize overall performance.

The agent's goal is to learn a set of optimal policies Π⋆ = {π⋆ θ(1) , . . . , π⋆ θ(Tmax ) } with corresponding parameters Θ⋆ = {θ(1)⋆ , . . . , θ(Tmax)⋆}. Since tasks belong to different domains, the dimension of the parameter vectors will vary, with θ(t) ∈ Rd(t) where d(t) is the dimension of the state space X (t) . At any time, the agent may be evaluated on any previous task, and so must strive to optimize its learned policies for all tasks Z (1) , . . . ,Z (T) , where T = ∑Gg=1 |G(g) | denotes the number of tasks seen so far (1 ≤ T ≤ Tmax) and G is the number of groups seen so far (1 ≤ G ≤ Gmax).

5 Cross-Domain Lifelong RL

This section develops our cross-domain lifelong RL approach. In order to share knowledge between the tasks, we assume that each task's policy parameters θ(t) ∈ Rd(t) for task t ∈ G(g) can be modeled as a sparse linear combination of latent components from knowledge base B(g) ∈ Rd(t)×k that is shared among all tasks in the group. Therefore, we have that θ(t) = B(g) s(t) , with sparse task-specific coefficients s(t) ∈ Rk for task t. The collection of all task-specific coefficients for tasks in G(g) is given by S(t∈G(g) ) ∈ Rk×|G(g) | .

Effectively, B(g) forms a k-component basis over the policy parameters for all tasks in G(g) , enabling the transfer of knowledge between tasks from this group. The taskspecific coefficients s(t) are encouraged to be sparse to ensure that each learned basis component captures a maximal reusable chunk of knowledge. This knowledge framework has been use successfully by a number of other MTL methods [Kumar & Daum´e III, 2012; Maurer et al., 2013; Ruvolo & Eaton, 2013] for transfer between tasks within a single task domain.

To support cross-domain transfer, we introduce a repository of knowledge L ∈ Rd×k that is shared among all tasks (including between task domains). This matrix L represents a set of latent factors that underly the set of group-specific basis matrices {B(1) , . . . ,B(Gmax) }. We introduce a set of group projection matrices Ψ(G(1) ) , . . . ,Ψ(G(Gmax) ) that map the shared latent factors L into the basis for each group of tasks. Therefore, we have that B(g) = Ψ(G(g) )L, where the group projection matrix Ψ(G(g) ) ∈ Rd(t)×d . With this construction, we see that each mapping Ψ(G(g) ) creates an intermediate knowledge space (i.e., B(g) ) that tailors the shared repository L into a basis that is suitable for learning tasks from group G(g) . These group-specific bases are coupled together via the Ψ mappings and the shared knowledge base L, facilitating transfer across task domains with different feature spaces. The group mappings Ψ's also serve to help avoid overfitting and ensure compactness of the basis, while maximizing transfer both between tasks within a group and across task groups. This construction is depicted in Figure 1.

5.1 The Cross-DomainMTLObjective

Under this shared knowledge framework, given a task t ∈ G(g) , its policy parameters θ(t) = Ψ(G(g) )Ls(t) , where Ψ(G(g) ) ∈ Rd(t)×d , L ∈ Rd×k , s(t) ∈ Rk , and k is the number of shared latent knowledge components. Therefore, to train optimal policies for all tasks, we must learn the shared knowledge base L, the group projections (the Ψ's), and the task-specific coefficients (the s(t) 's). We first examine this problem from a batch MTL standpoint, and then develop an efficient online algorithm for optimizing this MTL objective and enabling lifelong learning in the next section.

Given task groups G(1) , . . . , G(G) , we can represent our objective of learning T = ∑Gg=1 |G(g) | stationary policies while maximizing the amount of transfer between task policies as the problem of minimizing:

where θ(t) = Ψ(G(g) )Ls(t) and the L1 norm of s(t) is used to approximate the true vector sparsity. We employ regularization via the Frobenius norm || · ||F to avoid overfitting on both the shared knowledge base L and each of the group projections Ψ(G(1) ) , . . . ,Ψ(G(G) ) . Note that this objective is closely related to PG-ELLA [Bou Ammar et al., 2014], with the critical difference that it incorporates cross-domain transfer.

5.2 Online Solution to Cross-DomainMTL

Although Equation 2 allows for batch cross-domain MTL, the dependence on all available trajectories from all tasks (via J (θ(t) ) = ∫τ∈T(t) pθ(t) (τ ) R(t) (τ ) dτ ) make the batch approach unsuitable for learning tasks consecutively, since the learner requires all trajectories for acquiring a successful behavior. Here, we derive an approximate optimization algorithm that is more suitable for lifelong learning.

Standardizing the Objective

To derive the approximate optimization problem, we note that policy gradient methods maximize the lower bound of J (θ). In order to use Equation 2 for lifelong cross-domain transfer with policy gradients, we must first incorporate this lower bound into our objective function. Rewriting the error term in Equation 2 in terms of the lower bound yields

with ˜ θ(t) = Ψ(G(g) )Ls(t) . However, we can note that

Therefore, maximizing the lower bound of JL,θ (˜ θ(t)) is equivalent to the following minimization problem:

which can be plugged into Equation 3 in place of JL,θ (˜ θ(t)) to obtain the MTL policy gradients objective.

Approximate Learning Objective

To eliminate the dependence of the objective function on all available trajectories, we can approximate eT (·) by performing a second-order Taylor expansion of JL,θ ( ˜ θ(t)) around the optimal solution α(t) to Equation 4, following the technique used in PG-ELLA [Bou Ammar et al., 2014]. Note that attaining α(t) corresponds to solving the policy gradient problem of task t ∈ G(g) , which might be computationally expensive. Therefore, rather than using the above, we use an approximation acquired by performing a gradient step in task t: α(t) = θ + ηI -1∇˜ θ(t) JL,θ ( ˜ θ(t)), where I is the Fisher information matrix. The first derivative, needed for the second-order Taylor expansion, is given by:

Therefore, we have that

The second derivative of JL,θ ( ˜ θ(t)) is then:

Substituting the second-order Taylor expansion yields the following approximation of Equation 2:

where ||v||2A = vTAv, the constant term was suppressed as it has no effect on the minimization, and the linear term was ignored by construction since α(t) is a minimizer. Critically, we have eliminated the dependence on all available trajectories, making the problem suitable for an online MTL setting.

Learning the Policy

We fit the policy parameters in two steps. Upon observing a task t ∈ G(g) , we use gradient descent to update the shared repository L and the group projections Ψ(G(g) ) . The update rules, acquired by taking the derivative of Equation 5 with respect to L and Ψ(G(g) ) , can be written as:

where ηL and ηΨ(G(g) ) are the learning rates for the shared repository L and group projections Ψ(G(g) ) , respectively.

Having learned the shared representation L and group projections, task-specific coefficients s(t) are then determined by solving an instance of Lasso [Tibshirani, 1996] to encode α(t) in the basis given by Ψ(G(g) )L, leading to a task-specific policy π(t) = pθ(t)(a | x) where θ(t) = Ψ(G(g) )Ls(t) . For computationally demanding domains, we can update s(t) less frequently, instead allowing the policy to change by altering L and the Ψ's. The complete implementation of our approach is available on the authors' websites.

6 Theoretical Guarantees

This section shows that our approach becomes stable as the number of tasks and groups grow large. Detailed proofs and definitions are provided in an online appendix available on the authors' websites. First, we consider the one-group setting by defining the following expected loss: 1

where the expectation is over each task t in group G(g) according to the task's parameters (α(t) ,Γ(t) ), and l(·) is the per-task loss of encoding α(t) in the basis given by Ψ(G(g) )L. Proposition 1.

Proof. Here, we sketch the proof of Prop. 1. With l (L,ΨG(g) , s,α(t) ,Γ(t) ∣∣∣G(g)), we can show that

1 We super- or subscript variables with (|G(g) |) to denote the version of the variable learned from |G(g) | tasks in G(g) .

ˆh|G(g) | (Ψ(G(g) ) |G(g)) - ˆh|G(g) |-1 (Ψ(G(g) ) |G(g)) is Lipschitz with O ( 1 |G(g) | ). Further, given enough gradient steps, for Ψ(|G(g) |-1),(G(g) ) to minimize ˆh|G(g) |-1 (Ψ(G(g) ) |G(g)), it is clear that change in the loss can be upper bounded by 2λ ∣∣∣∣∣∣Ψ(|G(g) |),(G(g) ) - Ψ(|G(g) |-1),(G(g) ) ∣∣∣∣∣∣2 F . Combining the Lipschitz bound with previous facts concludes the proof.

Proof. First, we show that the sum of positive variations of the stochastic process u|G(g) | = ˆh|G(g) | (Ψ(|G(g) |),(G(g) ) |G(g)) are bounded by invoking a corollary of the Donsker theorem [Van der Vaart, 2000]. This result in combination with a theorem from Fisk [1965], allows us to show that ut(G(g) ) is a quasi-martingale that converges almost surely (a.s.). This fact along with a simple theorem of positive sequences allows us prove part 2 of the proposition. The final two parts (3 & 4) can be shown due to the equivalence of h and h|G(g) | as |G(g) | →∞.

Proposition 3. The distance between Ψ(|G(g) |),(G(g) ) and the set of h's stationary points converges a.s. to 0 as |G(g) | → ∞.

Proof. Both the surrogate ˆh|G(g) | and the expected cost h have gradients that are Lipschitz with constant independent of |G(g) |. This fact, in combination with the fact that ˆh|G(g) | and g converges a.s. as |G(g) | → ∞, completes the proof.

Next, we consider the loss of multiple groups:

Proposition 4.

Proof. This can be easily shown as the upper bound of ∑g (ˆh (Ψ(G(g) ) ∣∣∣G(g)) ·) is the sum of the bounds over

Proposition 5. With g(·) as the actual loss, we show: 1. ˆ g(G(g) ) (L) converges a.s. 2. ˆ g(G(g) ) (L) - g(G(g) ) (L) convergence a.s. to 0 3. ˆ g(G(g) ) (L) - g(G(g) ) (L) convergence a.s. to 0 4. g(G(g) ) (L) converges a.s.

Proof. This can be attained similarly to that of Prop. 2.

7 Empirical Results

We evaluated the ability of our approach to learn optimal control policies for multiple consecutive tasks from six different dynamical systems (each corresponding to one task domain):

Simple Mass (SM): The spring-mass-damper system is characterized by three parameters: spring and damping constants and the mass. The system's state is given by the position and velocity of the mass, which varies according to linear force. The goal is to control the mass to be in a specific state.

Double Mass (DM): The double spring-mass-damper has six parameters: two spring constants, damping constants, and masses. The state is given by the position and velocity of both masses. The goal is to control the first mass to a specific state, while only applying a linear force to the second.

Cart-Pole (CP): The dynamics of the inverted pendulum system are characterized by the cart's and pole's masses, the pole's length, and a damping parameter. The state is characterized by the cart's position and velocity, and the pole's angle and angular velocity. The goal is to balance the pole upright.

Double Cart-Pole (DCP): The DCP adds a second inverted pendulum to the CP system, with six parameters and six state features. The goal is to balance both poles upright.

Bicycle (Bike): The Bike model assumes a fixed rider, and is characterized by eight parameters. The goal is to keep the bike balanced as it rolls along the horizontal plane.

Helicopter (HC): This linearized model of a CH-47 tandem-rotor helicopter assumes horizontal motion at 40 knots. The main goal is to stabilize the helicopter by controlling the collective and differential rotor thrust.

For each of these systems, we created three different tasks by varying the system parameters to create systems with different dynamics, yielding 18 tasks total. These tasks used a reward function typical for optimal control, given by -√(xm - ˙x)T(xm - ˙x)-√aTmam where ˙x is the goal state. Each round of the lifelong learning experiment, one task t was chosen randomly with replacement, and task t's model was trained from 100 sampled trajectories of length 50. This process continued until all tasks were seen at least once.

We then compared the performance of cross-domain lifelong learning with PG-ELLA and standard policy gradients (PG), averaging results over ∼94 trials per domain (each of which contained ∼60 interleaved training rounds). As the base PG learner in all algorithms, we used Natural Actor Critic [Peters & Schaal, 2008]. All regularization parameters (the µ's) were set to e-5 , and the learning rates and latent dimensions were set via cross-validation over a few tasks.

Figure 2 shows the average learning performance on individual domains after this process of interleaved lifelong learning, depicting domains in which cross-domain transfer shows clear advantages over PG-ELLA and PG (e.g., DCP, HC), and an example domain where cross-domain transfer is less effective (CP). Note that even in a domain where cross-domain transfer provides little benefit, our approach achieves equivalent performance to PG-ELLA, showing that cross-domain transfer does not interfere with learning effectiveness. On all task domains except HC, our approach provides a significant increase in initial performance due to transfer (Figure 2(d)).

Cross-domain transfer provides significant advantages when the lifelong learning agent faces a novel task domain. To evaluate this, we chose the most complex of the task domains (helicopter) and trained the lifelong learner on tasks from all other task domains to yield an effective shared knowledge base L. Then, we evaluated the agent's ability to learn a new task from the helicopter domain, comparing the benefits of cross-domain transfer from L with PG-ELLA and PG (both of which learn from scratch on the new domain). Figure 3 depicts the result of learning on a novel domain, averaged over ten trials for all three HC tasks, showing the effectiveness of cross-domain lifelong learning in this scenario.

8 Conclusion

We have presented the first lifelong RL method that supports autonomous and efficient cross-domain transfer. This approach provides a variety of theoretical guarantees, and can learn effectively across multiple task domains, providing improved performance over single-domain methods.

Acknowledgements

This research was supported by ONR grant #N00014-11-10139 and AFRL grant #FA8750-14-1-0069. We thank the anonymous reviewers for their helpful feedback.