Using Task Features for Zero-Shot Knowledge Transfer in Lifelong Learning

1 Introduction

Transfer and multi-task learning (MTL) methods reduce the amount of experience needed to train individual task models by reusing knowledge from other related tasks. These techniques typically select the relevant knowledge to transfer by modeling inter-task relationships, based on training data for each task [Baxter, 2000; Ando & Zhang, 2005; Bickel et al., 2009; Maurer et al., 2013]. However, this process requires sufficient training data for each task to identify these relationships before knowledge transfer can succeed.

Consider instead the human ability to rapidly bootstrap a model for a new task, given only a high-level task description, before obtaining experience on the actual task. For example, viewing only the image on the box of a new Ikea chair, we can immediately identify previous related assembly tasks and begin formulating a plan to assemble the chair. In the same manner, an experienced inverted pole balancing agent may be able to predict the controller for a new pole given its mass and length, prior to interacting with the physical system.

These authors contributed equally to this work.

Inspired by this idea, we explore the use of high-level task descriptions to improve knowledge transfer between multiple machine learning tasks. We focus on lifelong learning scenarios [Thrun, 1996; Ruvolo & Eaton, 2013], in which multiple tasks arrive consecutively and the goal is to rapidly learn each new task by building upon previous knowledge. Although we focus on reinforcement learning (RL) tasks in this paper, our approach extends easily to regression and classification.

Our algorithm, Task Descriptors for Lifelong Learning (TaDeLL), encodes the task descriptions as feature vectors that identify each task, treating these descriptors as side information in addition to training data on the individual tasks. This idea of using task features for knowledge transfer has been explored previously by Bonilla et al. [2007] in an offline batch MTL setting, and more recently by Sinapov et al. [2015] in a computationally expensive method for estimating transfer relationships between pairs of tasks. In comparison, our approach operates online over consecutive tasks and is much more computationally efficient.

We use coupled dictionary learning to model the inter-task relationships between both the task descriptions and the individual task policies in lifelong learning. The coupled dictionary learning enforces the notion that tasks with similar descriptions should have similar policies, but still allows dictionary elements the freedom to accurately represent the different task policies. We connect the coupled dictionaries to the concept of mutual coherence in sparse coding, providing theoretical justification for why the task descriptors improve performance, and verify this improvement empirically.

In addition to improving the task policies, we show that the task descriptors enable the learner to accurately predict the policies for unseen tasks given only their descriptionthis process of learning without data is known as zero-shot learning. This capability is particularly important in the online setting of lifelong learning, enabling the system to accurately predict policies for new tasks through transfer, without requiring it to pause to gather training data on each task.

2 RelatedWork

Batch MTL [Caruana, 1997] methods often model the relationships between tasks to determine the knowledge to transfer [Baxter, 2000; Ando & Zhang, 2005; Bickel et al., 2009;

Lazaric & Ghavamzadeh, 2010; Maurer et al., 2013]. These techniques include modeling a task distance metric [BenDavid et al., 2007], using correlations to determine when transfer is appropriate [Wang et al., 2014], or building models based on nearest neighbors [Parameswaran & Weinberger, 2010]. More recently, MTL has been extended to a lifelong learning setting, in which tasks arrive consecutively for regression and classification [Ruvolo & Eaton, 2013] and for reinforcement learning [Bou Ammar et al., 2014; 2015]. However, all of these methods require training data for each task in order to assess their relationships and determine the knowledge to transfer.

Instead of relying solely on the tasks' training data, several works have explored the use of high level task descriptors to model the inter-task relationships in MTL and transfer learning settings. Task descriptors have been used in combination with neural networks [Bakker & Heskes, 2003] to define a task-specific prior or to control the gating network between individual task clusters. Bonilla et al. [2007] explore similar techniques for multi-task kernel machines, using task features in combination with the data for a gating network over individual task experts or to augment the original task training data. These papers focus on multi-task classification and regression in batch settings where the system has access to the data and features for all tasks, in contrast to our study of task descriptors for lifelong learning over consecutive RL tasks.

In the work most similar to ours, Sinapov et al. [2015] use task descriptors to estimate the transferability between each pair of tasks for transfer learning. Given the descriptor for a new task, they identify the source task with the highest predicted transferability, and use that source task for a warm start in RL. Though effective, their approach is computationally expensive, since they estimate the transferability for every task pair through repeated simulation. Their evaluation is also limited to a transfer learning setting, and they do not consider the effects of transfer over consecutive tasks or updates to the transferability model, as we do in the lifelong setting.

Our work is also related to the Simple Zero-Shot Learning (Simple ZSL) method by Romera-Paredes and Tor [2015], which learns a multi-class linear model, and factorizes the linear model parameters, assuming the descriptors are coefficients over a latent basis to reconstruct the models. Our approach assumes a more flexible relationship: that both the model parameters and task descriptors can be reconstructed from separate latent bases that are coupled together through their coefficients. In comparison to our lifelong learning approach, Simple ZSL operates in an offline multi-class setting.

3 Background

3.1 Reinforcement Learning

A reinforcement learning (RL) agent must select sequential actions in an environment to maximize its expected return. An RL task is typically formulated as a Markov Decision Process (MDP) hX↪A↪ P↪R↪ γi, where X is the set of states, and A is the set of actions that the agent may execute, P : X ⇥A⇥X ! [0↪ 1] is the state transition probability describing the systems dynamics, R : X ⇥ A ⇥ X ! R is the reward function, and γ 2 [0↪ 1) is the discount assigned to rewards over time. At time step h, the agent is in state xh 2 X and chooses an action a 2 A according to policy ⇡ : X ⇥A7! [0↪ 1], which is represented as a function defined by a vector of control parameters ✓ 2 Rd . The agents then receives reward rh according to R and transitions to state xh+1 according to P. This sequence of states, actions, and rewards is given as a trajectory ⌧ = ¶(x1↪ a1↪ r1)↪ ▷ ▷ ▷ ↪ (xH↪ aH↪ rH)♦ over a horizon H. The goal of RL is to find the optimal policy ⇡⇤ with parameters ✓⇤ that maximizes the expected reward. However, learning an individual task still requires numerous trajectories, motivating the use of transfer to reduce the number of interactions with the environment.

Policy Gradient (PG) methods [Sutton et al., 1999], which we employ as our base learner, are a class of RL algorithms that are effective for solving high dimensional problems with continuous state and action spaces, such as robotic control [Peters & Schaal, 2008]. The goal of PG is to optimize the expected average return: J(✓) = E h 1H PHh=1 rh i = RT p✓(⌧)R(⌧)d⌧ , where T is the set of all possible trajectories, the average reward on trajectory ⌧ is given by R(⌧) = 1H PHh=1 rh, and p✓(⌧) = P0(x1) QHh=1 p(xh+1 ♣ xh↪ ah) ⇡(ah ♣ xh) is the probability of ⌧ under an initial state distribution P0 : X 7! [0↪ 1]. Most PG methods (e.g., episodic REINFORCE [Williams, 1992], PoWER [Kober & Peters, 2009], and Natural Actor Critic [Peters & Schaal, 2008]) optimize the policy by maximizing a lower bound on J (✓), comparing trajectories generated by ⇡✓ against those generated by a new candidate policy ⇡˜ ✓ . For details, see Kober & Peters [2009].

3.2 LifelongMachine Learning

In a lifelong learning setting [Thrun, 1996; Ruvolo & Eaton, 2013], the learner faces multiple, consecutive tasks and must rapidly learn each new task by building upon its previous experience. The learner may encounter a previous task at any time, and so must optimize performance across all tasks seen so far. A priori, the agent does not know the total number of tasks Tmax, the task distribution, or the task order.

At time t, the lifelong learner encounters task Z (t) . In this paper, each task Z (t) is specified by an MDP hX (t) ↪A(t) ↪ P (t) ↪R(t) ↪ γ (t) i, but the lifelong learning setting and our approach can equivalently handle classification or regression tasks. The agent will learn each task consecutively, acquiring training data (i.e., trajectories) in each task before advancing to the next. The agent's goal is to learn the optimal policies ¶⇡⇤ ✓(1) ↪ ▷ ▷ ▷ ↪ ⇡⇤ ✓(T)♦ with corresponding parameters ¶✓(1) ↪ ▷ ▷ ▷ ↪ ✓(T) ♦, where T is the number of unique tasks seen so far (1  T  Tmax). Ideally, knowledge learned from previous tasks ¶Z (1) ↪ ▷ ▷ ▷ ↪ Z (T-1) ♦ should accelerate training and improve performance on each new task Z (T) . Also, the lifelong learner should scale effectively to large numbers of tasks, learning each new task rapidly from minimal data.

The Efficient Lifelong Learning Algorithm (ELLA) [Ruvolo & Eaton, 2013] and PG-ELLA [Bou Ammar et al., 2014] were developed to operate in this lifelong learning setting for classification/regression and RL tasks, respectively. Both approaches assume the parameters for each task model can be factorized using a shared knowledge base L, facilitating transfer between tasks. Specifically, the model parameters for task Z (t) are given by ✓(t) = Ls(t) , where L 2 Rd⇥k is the shared basis over the model space, and s(t) 2 Rk are the sparse coefficients over the basis. This factorization has been effective for transfer in both lifelong and multi-task learning [Kumar & Daum´e, 2012; Maurer et al., 2013].

Under this assumption, the MTL objective for PG is:

where S = [s(1) · · · s(T) ] is the matrix of sparse vectors, J (✓(t) ) is the PG objective for task Z (t) , k · kF is the Frobenius norm, the L1 norm is used to approximate the true vector sparsity of s(t) , and θ and λ are regularization parameters. To solve this objective in a lifelong learning setting, Bou Ammar et al. [2014] approximate the multi-task objective by first substituting in the lower-bound to the PG objective, then taking a second-order Taylor expansion to approximate the objective around an estimate ↵(t) 2 Rd of the single-task policy parameters for each task Z (t) , and updating only the coefficients s(t) for the current task at each time step. This process reduces the MTL objective to the problem of sparse coding the single-task policies in the shared basis L, and enables S and L to be solved efficiently by the following online update rules that constitute PG-ELLA [Bou Ammar et al., 2014]:

where kvk2A = v>Av, the symbol ⌦ denotes the Kronecker product, Γ(t) is the Hessian of the PG lower bound on J (↵(t) ), Im is the m⇥m identity matrix, A is initialized to a kd ⇥ kd zero matrix, and b 2 Rkd is initialized to zeros.

Though effective for lifelong learning, this approach requires significant training data to estimate the policy for each new task before the learner can solve it. We eliminate this restriction by incorporating task descriptors into lifelong learning, enabling zero-shot transfer to new tasks.

4 Task Descriptors

While most MTL and lifelong learning methods use task training data to model inter-task relationships, high level descriptions can describe task differences. For example, in multi-task medical domains, patients are often grouped into tasks by demographic data and disease presentation [Oyen & Lane, 2012]. In control problems, the dynamical system parameters (e.g., the spring, mass, and damper constants in a spring-mass-damper system) describe the task. Descriptors can also be derived from external sources, such as Wikipedia [Pennington et al., 2014]. Such task descriptors have been used extensively for zero-shot learning [Palatucci et al., 2009; Socher et al., 2013].

Formally, we assume that each task Z (t) has an associated descriptor m(t) that is given to the learner upon first presentation of the task. The learner has no knowledge of future tasks, or the distribution of task descriptors. The descriptor is represented by a feature vector φ(m(t) ) 2 Rdm , where φ(·) performs feature extraction and (possibly) a non-linear basis transformation on the features. We make no assumptions on the uniqueness of φ(m(t) ), although in general tasks will have different descriptors. 1 In addition, each task also has associated training data X(t) to learn the model; in the case of RL tasks, the data consists of trajectories that are dynamically acquired by the agent through experience in the environment.

5 Lifelong Learning with Task Descriptors

We incorporate task descriptors into lifelong learning via sparse coding with a coupled dictionary, enabling the descriptors and learned policies to augment each other. Although we focus on RL tasks, our method can easily be adapted to classification or regression, as described in the Appendix. 2

5.1 Coupled Dictionary Optimization

As described previously, many multi-task and lifelong learning approaches have found success with factorizing the policy parameters ✓(t) for each task as a sparse linear combination over a shared basis: ✓(t) = Ls(t) . In effect, each column of the shared basis L serves as a reusable policy component representing a cohesive chunk of knowledge. In lifelong learning, the basis L is refined over time as the system learns more tasks. The coefficient vectors S = [s(1) ▷ ▷ ▷ s(T) ] encode the task policies in this shared basis, providing an embedding of the tasks based on how their policies share knowledge.

We make a similar assumption about the task descriptorsthat the descriptor features φ(m(t) ) can be linearly factorized 3 using a latent basis D 2 Rdm⇥k over the descriptor space. This basis captures relationships among the descriptors, with coefficients that similarly embed tasks based on commonalities in their descriptions. From a co-view perspective [Yu et al., 2014], both the policies and descriptors provide information about the task, and so each can augment the learning of the other. Each underlying task is common to both views, and so we seek to find task embeddings that are consistent for both the policies and their corresponding task descriptors. We can enforce this by coupling the two bases L and D, sharing the same coefficient vectors S to reconstruct both the policies and descriptors. Therefore, for task Z (t) ,

To optimize the coupled bases L and D during the lifelong learning process, we employ techniques for coupled dictionary optimization from the sparse coding literature [Yang

1 This raises the question of what descriptive features to use, and how task performance will change if some descriptive features are unknown. We explore these issues empirically in the Appendix.

2 The online appendix is available on the third author's website.

3 This is potentially non-linear w.r.t m(t) , since φ can be non-linear.

Algorithm 1 TaDeLL (k, λ, θ)

1: T 0 2: L RandomMatrixd↪k↪ D RandomMatrixm↪k 3: while some task (Z (t) ↪ φ(m(t) )) is available do 4: if isNewTask(Z (t) ) then 5: T T +1 6: T(t) sampleRandomTrajectories(Z (t) ) 7: else 8: T(t) sampleTrajectories(Z (t) , ⇡↵(t) ) 9: end if 10: Compute ↵(t) and Γ(t) fromT(t) 11: s(t) argmins ∥∥β(t) -Ks∥∥2A(t) +θksk1 12: L updateL(L↪ s(t) ↪↵(t) ↪Γ(t) ↪ λ) 13: D updateD(D↪ s(t) ↪ φ(m(t) )↪ ⇢Idm ↪ λ) 14: for t 2 ¶1↪ ▷ ▷ ▷ ↪ T♦ do: ✓(t) Ls(t) 15: end while

et al., 2010], which optimizes the dictionaries for multiple feature spaces that share a joint sparse representation. This notion of coupled dictionary learning has led to high performance algorithms for image super-resolution [Yang et al., 2010], allowing the reconstruction of high-res images from low-res samples, and for multi-modal retrieval [Zhuang et al., 2013] and cross-domain retrieval [Yu et al., 2014].

Given the factorization in Eq. 6, we can re-formulate the multi-task objective (Eq. 1) for the coupled dictionaries as

To solve Eq. 7 online, we approximate J (·) by a secondorder Taylor expansion around ↵(t) , the minimizer for the PG lower bound of J(·) (i.e., ⇡↵(t) is the single-task policy for Z (t) based on the observed trajectories), following Bou Ammar et al. [2014]. This simplifies Eq. 7 to

This objective can now be solved efficiently online, as a series of per-task update rules given in Algorithm 1. L and D are updated independently using Equations 3-5, following a recursive construction based on an eigenvalue decomposition.

The complete implementation of our approach is available on the third author's website.

Algorithm 2 Zero-Shot Transfer to a New Task Z (tnew )

• 1: Inputs: task descriptor m(tnew) , learned bases L andD 2: ˜ s(tnew) argmins ∥∥φ(m(tnew) ) -Ds∥∥22 +θksk1 3: ˜ ✓(tnew) L˜ s(tnew ) 4: Return: ⇡˜ ✓(tnew )

5.2 Zero-Shot Transfer Learning

In a lifelong setting, when faced with a new task, the agent's goal is to learn an effective policy for that task as quickly as possible. At this stage, previous multi-task and lifelong learners incurred a delay before they could produce a decent policy, since they needed to acquire data from the new task in order to identify related knowledge and train the new policy.

Incorporating task descriptors enables our approach to predict a policy for the new task immediately, given only the descriptor. This ability to perform zero-shot transfer is enabled by the use of coupled dictionary learning, which allows us to observe a data instance in one feature space (i.e., the task descriptor), and then recover its underlying latent signal in the other feature spaces (i.e., the policy parameters) using the dictionaries and sparse coding [Yang et al., 2010].

Given only the descriptor m(tnew) for a new task Z (tnew) , we can estimate the embedding of the task in the latent descriptor space via LASSO on the learned dictionary D:

Since the estimate given by s(tnew) also serves as the coefficients over the latent policy space L, we can immediately predict a policy for the new task as: ˜ ✓(tnew ) = L˜ s(tnew) . This zero-shot transfer learning procedure is given as Algorithm 2.

5.3 Theoretical Analysis

This section discusses why incorporating task descriptors through coupled dictionaries can improve performance of the learned policies and enable zero-shot transfer to new tasks. In the Appendix2 , we also prove the convergence of TaDeLL. A full sample complexity analysis is beyond the scope of this paper, and, indeed, remains an open problem for zero-shot learning [Romera-Paredes & Torr, 2015].

To analyze the policy improvement, since the policy parameters are factored as ✓(t) = Ls(t) , we proceed by showing that incorporating the descriptors through coupled dictionaries can improve both L and S. In this analysis, we employ the concept of mutual coherence, which has been studied extensively in the sparse recovery literature. Mutual coherence measures the similarity of a dictionary's elements as M(Q) =max1i6=jn ⇣ ♣q> i qj♣ kqik2kqjk2 ⌘ 2 [0↪ 1], where qi is the ith column of a dictionary Q 2 Rd⇥k . If M(Q) = 0, then Q is an invertible orthogonal matrix and so sparse recovery can be solved directly by inversion; M(Q) = 1 implies that Q is not full rank and a poor dictionary. Intuitively, low mutual coherence indicates that the dictionary's columns are considerably different, and thus such a "good" dictionary can represent many different policies, potentially yielding more knowledge transfer. This intuition is shown in the following:

Therefore, an L with low mutual coherence would lead to more stable solutions of the ✓(t) 's against inaccurate singletask estimates of the policies (the ↵(t) 's). We next show that our approach likely lowers the mutual coherence of L.

TaDeLL alters the problem from training L to training the coupled dictionaries L and D (contained in K). Let s⇤(t) be the solution to Eq. 1 for task Z (t) , which is unique under sparse recovery theory, so ∥∥s⇤(t)∥∥0 remains unchanged for all tasks. Theorem 5.1 implies that, if M(K) < M(L), coupled dictionary learning can help with a more accurate recovery of the s(t) 's. To show this, we note that Eq. 7 can also be derived as a result of an MAP estimate from a Bayesian perspective, enforcing a Laplacian distribution on the s(t) 's and assuming L to be a Gaussian matrix with elements drawn i.i.d: lij ⇠ N(0↪ σ2 ). When considering such a random matrix Q 2 Rd⇥k , Donoho & Huo [2001] proved that asymptotically M(Q) / q log(dk) d as d ! 1. Using this as an estimate for M(L) and M(K), since incorporating task descriptors increases d, most likely M(K) < M(L), implying that TaDeLL learns a superior dictionary. Moreover, if M(D)  M(L), the theorem implies we can use D alone to recover the task policies through zero-shot transfer.

To show that task features can also improve the sparse recovery, we rely on the following theorem about LASSO:

Theorem 5.2. [Negahban et al., 2009] Let s⇤ be a unique solution to the system ✓ = Qs with ksk0 = K and Q 2 Rd⇥k . If s+ is the LASSO solution for the system from noisy observations, then with high probability: ks⇤ - s+k2  c0qK log k d , where the constant c0 2 R+ depends on properties of the linear system and observations.

This theorem shows that the error reconstruction for LASSO is proportional to 1 pd . When we incorporate the descriptor through β(t) , the RHS denominator increases from d to (d + dm) while K and k remain constant, yielding a tighter fit. Therefore, task descriptors can improve learned dictionary quality and sparse recovery accuracy. To ensure an equivalently tight fit for s(t) using either policies or descriptors, Theorem 5.2 suggests it should be that dm ≥ d to ensure that zero-shot learning yields similarly tight estimates of s(t) .

Computational Complexity Each update begins with one PG step to update ↵(t) and Γ(t) at a cost of O(⇠(d↪ nt)), where ⇠() depends on the base PG learner and nt is the number of trajectories obtained for task Z (t) . The cost of updating L and s(t) alone is O(k2d3 ) [Ruvolo & Eaton, 2013], and so the cost of updating K through coupled dictionary learning is O(k2 (d + dm)3 ). This yields an overall per-update cost of O(k2 (d + dm)3 + ⇠(d↪ nt)), which is independent of T .

6 Experiments

We evaluated our method on learning control policies for three benchmark systems and an application to quadrotors.

6.1 Benchmark Dynamical Systems

Spring Mass Damper (SM) The SM system is described by three parameters: the spring constant, mass, and damping constant. The system's state is given by the position and velocity of the mass. The controller applies a force to the mass, attempting to stabilize it to a given position.

Cart Pole (CP) The CP system involves balancing an inverted pendulum by applying a force to the cart. The system is characterized by the cart and pole masses, pole length, and a damping parameter. The states are the position and velocity of the cart and the angle and rotational velocity of the pole.

Bicycle (BK) This system focuses on keeping a bicycle balanced upright as it rolls along a horizontal plane at constant velocity. The system is characterized by the bicycle mass, xand z-coordinates of the center of mass, and parameters relating to the shape of the bike (the wheelbase, trail, and head angle). The state is the bike's tilt and its derivative; the actions are the torque applied to the handlebar and its derivative.

6.2 Methodology

In each domain we generated 40 tasks, each with different dynamics, by varying the system parameters. The reward for each task was taken to be the distance between the current state and the goal. For lifelong learning, tasks were encountered consecutively with repetition, and learning proceeded until each task had been seen at least once. We used the same random task order between methods to ensure fair comparison. The learners sampled trajectories of 100 steps, and the learning session during each task presentation was limited to 30 iterations. For MTL, all tasks weres presented simultaneously. We used Natural Actor Critic [Peters & Schaal, 2008] as the base learner for the benchmark systems and episodic REINFORCE [Williams, 1992] for quadrotor control. We chose k and the regularization parameters independently for each domain to optimize the combined performance of all methods on 20 held-out tasks, and set ⇢ = mean(diag(⇢(t) )) to balance the fit to the descriptors and the policies. We measured learning curves based on the final policies for each of the 40 tasks, averaging results over seven trials. The system parameters for each task were used as the task descriptor features φ(m); we also tried several non-linear transformations as φ(·), but found the linear features worked well.

6.3 Results on Benchmark Systems

Figure 1 compares our TaDeLL approach for lifelong learning with task descriptors to 1.) PG-ELLA [Bou Ammar et al., 2014], which does not use task features, 2.) GO-MTL [Kumar & Daum´e, 2012], the MTL optimization of Eq. 1, and 3.) single-task learning using PG. For comparison, we also performed an offline MTL optimization of Eq. 7 via alternating optimization, and plot the results as TaDeMTL. The shaded regions on the plots denote standard error bars.

We see that task descriptors improve lifelong learning on every system, even driving performance to a level that is unachievable from training the policies from experience alone via GO-MTL in the SM and BK domains. The difference between TaDeLL and TaDeMTL is also negligible for all domains except CP (which had very diverse tasks), demonstrating the effectiveness of our online optimization.

Figure 3: Zero-shot transfer to new tasks. Figure (a) shows the initial "jumpstart" improvement on each task domain; Figures (b)-(d) depict the result of using zero-shot policies as warm start initializations for PG. (Best viewed in color.)

Figure 3 shows that task descriptors are effective for zeroshot transfer to new tasks. We generated an additional 40 tasks for each domain to measure zero-shot performance, averaging results over these new tasks. Figure 3a shows that our approach improves the initial performance (i.e., the "jumpstart" [Taylor & Stone, 2009]) on new tasks, outperforming Sinapov et al. [2015]'s method and single-task PG, which was allowed to train on the task. We attribute the especially poor performance of Sinapov et al. on CP to the fact that the CP policies differ substantially; in domains where the source policies are vastly different from the target policies, Sinapov et al.'s algorithm does not have an appropriate source to transfer. Their approach is also much more computationally expensive (quadratic in the number of tasks) than our approach (linear in the number of tasks), as shown in Figure 2; details of the runtime experiments are included in the Appendix2 . Figures 3b-3d show that the zero-shot policies can be used effectively as a warm start initialization for a PG learner, which is then allowed to improve the policy.

6.4 Application to Quadrotor Control

We also applied our approach to the more challenging domain of quadrotor control, focusing on zero-shot transfer to new stability tasks. To ensure realistic dynamics, we use the model of Bouabdallah and Siegwart [2005], which has been verified on physical systems. The quadrotors are characterized by three inertial constants and the arm length, with their state consisting of roll/pitch/yaw and their derivatives.

Figure 4 shows the results of our application, demonstrating that TaDeLL can predict a controller for new quadrotors through zero-shot learning that has equivalent accuracy to PG, which had to train on the system. As with the benchmarks, TaDeLL is effective for warm start learning with PG.

7 Conclusion

We proposed a coupled dictionary method for incorporating task descriptors into lifelong learning, showing that descriptors improve learned policy performance, and enable us to predict policies for new tasks before observing training data. Experiments demonstrate that our method outperforms other approaches on dynamical control problems, and requires substantially less computational time than similar methods.

Acknowledgments

This research was supported by ONR grant #N00014-11-10139 and AFRL grant #FA8750-14-1-0069. We would like to thank the anonymous reviewers for their helpful feedback.