Independent Skill Transfer for Deep Reinforcement Learning

1 Introduction

Deep reinforcement learning (DRL) has wide applications in various challenging fields, such as real-world visual navigation [Zhu et al., 2017], playing games [Silver et al., 2016] and robotic controls [Schulman et al., 2015]. However, conventional algorithms are incapable to deal with complex environments with quite difficult tasks and extremely sparse reward [Kulkarni et al., 2016]. Inspired by the human intelligence that can explore the environment by themselves and learn various skills to significantly improve their ability and accomplish tasks, multi-skill DRL has been proposed as a potential solution to handle tasks in complex environments [Eysenbach et al., 2018]. A significant breakthrough of muti-skill learning is the development of autonomous skill discovery, which can acquire multiple skills autonomously without extrinsic reward by maximizing an information theoretic objective [Gregor et al., 2016] [Singh et al., 2019]. By transferring pre-trained skills, the learning process in a new environment can be greatly shorten and the efficiency is thus enhanced.

∗Corresponding author

In this paper, skill transfer is to learn primitive skills in a source environment without extrinsic reward and reutilize them in a target environment, which can acquire a higher learning efficiency than learning from scratch. Skill transfer can greatly accelerate the learning process in a target environment because the learned primitive skills provides reusable abstraction of the source environment [Sahni et al., 2017]. However, with the existing direct skill transfer, there are two problems resulting in the low learning efficiency:

• Strong correlation: We analyze the statistical characteristics of primitive skills that are generated from the existing skill discovery methods [Eysenbach et al., 2018] [Sharma et al., 2019]. By using principal components analysis (PCA) [Bryant and Yarnold, 1995] on actions of primitive skills, Figure 1 shows the eigenvalue and the corresponding percentage of principal components in primitive skills. It is observed that the actions generated by distinct primitive skills are strong correlated with each other, which indicates that the amount of skills can be reduced by eliminating this correlation so that the skill dimension is thus decreased.
• Unbalance of skill discovery and transfer: In the skill discovery, the agent learns each of primitive skills separately in a source environment. However, in the skill transfer, all primitive skills are combined to instruct the

agent in a target environment [Peng et al., 2019]. As noticed, the primitive skill in skill discovery and skill transfer play a distinct role. A more balanced scheme might improve the performance of skill transfer.

To overcome these two problems above, we propose to learn independent skills for efficient skill transfer, where the learned primitive skills with strong correlation are decomposed into independent skills. More concretely, Figure 2 shows the process of independent skill transfer (IST). Diverse primitives skills are firstly cultivated in a source environment. By taking actions to represent such primitive skills, the agent then decomposes these primitive skills to acquire independent skills, where independent component analysis (ICA) [Hyv¨arinen and Oja, 2000] is employed on those primitive skill's actions. Finally, the agent transfers independent skills into new practical skills in a target environment.

On the one hand, our proposed IST method is able to reduce the dimension of skills and enhance the efficiency of skill transfer. On the other hand, each of primitive skills is the combination of all independent skills, which is balanced with the combined practical skill in Figure 2. Therefore, the advantage of our proposed IST can be listed as follows:

• The correlation between skills can be largely deducted and a lower dimension is thus obtained to enhance the efficiency of skill transfer.
• Combination of independent skills take effects in both skill discovery and transfer, where transferring independent skills is more essential.
• Independent skills are task-independent, which can be transferred to a variety of practical skills in a target environment.

The contribution of this work is summarized as follow. First, we propose to learn low-dimension independent skills from primitive skills. Secondly, we propose to transfer independent skills to practical skills in target environments, which exhibits a higher efficiency and better generalization than conventional methods. Finally, various experiments are conducted to demonstrate the effectiveness of IST method.

2 RelatedWork

In recent years, skill discovery and transfer has gained more and more attention from researchers who work in DRL.

Skill Discovery. Information theory has been widely applied in the development of multiple skills in DRL. In order to enable the agent to own skills to explore the environment, the VIM [Mohamed and Rezende, 2015] adds the relative entropy between states and actions into the reward function to broaden the observation of the agent. Soft actor-critic (SAC) [Haarnoja et al., 2018a] discovers a policy function with maximum entropy to gather useful skills for the agent. Moreover, an important work, i.e. the DIAYN [Eysenbach et al., 2018], also maximizes the mutual information between skills and states to ensure that states are used to distinguish skills.

Skill transfer. A popular method of skill transfer is to learn a hierarchical policy for skill reutilization. In the pre-train stage, a collection of primitive skills are learned based on skill discovery methods, where each skill is encouraged to specialize distinct observations [Coros et al., 2009] [Frans et al., 2017]. While in transferring the primitive skills, a meta controller is trained to select primitive skills depending on the specific high-level task [Hausknecht and Stone, 2015]. Even though existing methods on skill transfer are elaborative and remarkable, several drawbacks remain to be solved. One of significant problems is the low transfer efficiency since the integration of primitive skills fails to consider their dependencies on each other [Peters et al., 2017].

Independent component analysis (ICA). ICA attempts to decompose a multivariate signal into independent nonGaussian signals. Preprocessing steps for ICA involves centering and dimensionality reduction, which can be achieved by PCA. Existing works on ICA include kernel-independent component analysis [Bach and Jordan, 2002], infomax [Bell and Sejnowski, 1995] and FastICA [Oja and Yuan, 2006]. In this work, we consider to employ FastICA to decompose primitive skills into independent skills, which allows much easier generalization and transfer [Bengio, 2017], and can greatly enhance the transfer efficiency.

3 Preliminary

3.1 Soft Actor-critic

A standard reinforcement learning framework is characterized by observation space S, action space A, reward function r, transition function T and discount factor γ, which formulates a five-tuple < S,A, r, T, γ >. At each step t, an agent observes the current state st ∈ S, and executes an action a ∈ A according to the policy function ϖφ(a|s) : S → A. In this paper, we use the SAC [Haarnoja et al., 2018b] algorithm to update the value function Q(s, a) and the policy function. ϖφ(a|s) is updated to maximize the following cumulative reward J(φ) (i.e. φ∗ = argmax J(φ))

where N denotes the spherical Gaussian distribution; f (ϵ, st) can be obtained via reparameterization trick; the entropy term -Eat∼ϖφ logϖφ(a|s) multiplied by temperature parameter β is introduced to control the encouragement of exploration.

3.2 Skill Discovery

The skill space Z is introduced to the framework of standard reinforcement learning to diversify the learning process of agents. Different skills π(a|s, z) : S × Z → A can alter the current observation by the agent in a consistent manner.

In the recent work DIAYN [Eysenbach et al., 2018], the authors propose to use intrinsic reward for skill discovery. In each step t, Q(st, z, at) : S×Z×A → R and π(at|st, z) are updated based on SAC, and a discriminator network q(z|st) is employed to learn the posterior distribution of each skill given st, where its parameters are updated via stochastic gradient descent (SGD). To generate a variety of different skills, DIAYN employs the reward function given by

where the prior p(z) is often selected as uniform distribution.

4 Learn Independent Skills (LIS)

In this paper, we use policy networks ˆπθ(a|s, ˆ z), ˆ z ∈ ˆ Z to denote independent skills. For the convenience of specification, we define the actions generated by independent skill as independent action, and those produced by primitive skill as primitive action. Furthermore, our proposed LIS is based on the following three assumptions:

• Primitive actions are the linear combination of independent actions.
• On the same observation, the actions generated by distinct independent skill are independent with each other, which reflects the independence between independent skills.
• Without loss of generality, the number of independent skill | ˆ Z| is no more than the number of primitive skills |Z|.

In brief, the aim of LIS is to learn finite ˆπθ(a|s, ˆ z), ˆ z ∈ ˆ Z, such that ∀s ∈ S, a ∼ π(a|s, z) is the linear combination of ˆa ∼ ˆπθ(a|s, ˆ z). As shown in Figure 3, we first sample subset ˜ S from the observation space S. Secondly, we sample action Az based on primitive skill π(a|s, z) for z ∈ Z. We then convert the primitive actions Az to independent actions ˆAz through ICA. Finally, we utilize ˜ S and ˆAz to learn independent skills ˆπθ(a|s, ˆ z) for ˆ z ∈ ˆ Z via supervised learning.

4.1 Collection of Observation and Action

It is inappropriate to sample observations from the whole observation space S that is extremely large and continuous. Hence, a strategy for collecting observation and action is proposed in this paper, where we sample a key subset ˜ S ⊂ S from the observation space.

Different skills enable an agent to have multiple, stable and finite state transitions, which are considered as key subset of observation space S in this paper. Specifically, for each primitive skill π(a|s, z), we sample L trajectories τz,1, τz,2, ..., τz,L independently, where each τz,i = {sz,i,1, az,i,1, sz,i,2, az,i,2, ..., sz,i,Ti } for 1 ≤ i ≤ L denotes

Algorithm 1 Learn independent skills (LIS)

Input:π(a|s, z), z ∈ Z

Output:ˆπθ(a|s, ˆ z), ˆ z ∈ ˆ Z

• 1: Sample ˜ S1, ˜ S2, ..., ˜ S|Z| via Eq. (3).
• 2: Sample A1,A2, ...,A|Z| via Eq. (4), and transform to A1, A2, · · · , A|Z|. //--------- ICA----------//
• 3: Calculate WP via PCA, and whiten A = [A1,A2, · · · ,A|Z|].
• 4: while not converged do
• 5: CalculateWI according to Eq. (6) and (7).
• 6: end while
• 7: Calculate ˆ A1, ˆ A2, ..., ˆ A| ˆ Z| via Eq. (5), and reconstruct to

• 8: for s ∈ ˜ S, ˆ z ∈ ˆ Z do
• 9: Calculate µs,ˆ z and σs,ˆ z .
• 10: min θ {DKL[ ˆπθ(ˆa|s, ˆ z)|| ˆ p(ˆa|µs,ˆ z , σs,ˆ z )]}.
• 11: end for
• 12: return ˆπθ(a|s, ˆ z), ˆ z ∈ ˆ Z

the trajectory in each episode. The observation set of skill z consists of all the states in τz,i for 1 ≤ i ≤ L, given by

Thus, the key subset ˜ S of observation space can be expressed as ˜ S = ˜ S1 ∪ ˜ S2 ∪ ... ∪ ˜ S|Z|.

In the action collection, for each skill π(a|s, z), we sample K actions on each of the observations in ˜ S, and reserve all the actions in the following matrix Az

where it is worth noting that partial actions of π(a|s, z) on ˜ Sz can be directly obtained from trajectories τz,1, τz,2, ..., τz,L.

As the mixed signal for ICA, matrix Az has to be flattened 1 , i.e. Az = flatten(Az). After the processing of ICA, we get independent components ˆ Aˆ z from Az. Then we reconstruct ˆ Aˆ z to matrix ˆAz by adopting the inverse operation.

4.2 Generation of Independent Actions

In this subsection, the primitive action Az, z ∈ Z is regarded as mixed signal to generate independent actions ˆ Aˆ z, ˆ z ∈ ˆ Z. Specifically, we aim to find a | ˆ Z|×|Z| matrixWP and | ˆ Z|× | ˆ Z| full-rank matrixWI, such that

1 A general flattening like column-based or row-based conversion works for the transformation of n×K| ˜ S| matrix Az into nK| ˜ S| × 1 vector Az, where n is the dimension of a. Here, column-based transformation is applied.

where ˆ A1, ˆ A2, ..., ˆ A| ˆ Z| are independent with each other.

In this paper, we use PCA [Wold et al., 1987] to estimate WP , where the number of independent skills | ˆ Z| can be achieved by clustering the eigenvalues of cross-correlation matrix of Az. We take the eigenvalues in Figure 1 as an example: for the case of 6 primitive skills, | ˆ Z| = 3 is reasonable since more than 98% component of primitive actions can be represented by three independent components.

To estimate WI , we employ the FastICA [Hyv¨arinen and Oja, 2000], where the independent signals are actually independent actions. The details are given as follows,

• First of all, the primitive actions A = [A1, · · · ,A|Z|] has to be whitened so that E[AAT ] = I.
• Secondly, we estimate WI = [w1,w2, · · · ,w|Z|] by calculating each of its columns, separately. In each iteration of ICA, we maximize the independence of output signal via updating wi by

for 1 ≤ i ≤ | ˆ Z|, where g(x) = 1 1+e-x , g′ (x) = -e-x 1+e-x , and we orthogonalize WI by

After the algorithm converges, the independent actions ˆ Aˆ z for ˆ z ∈ ˆ Z can be obtained according to Eq. (5).

4.3 Generation of Independent Skills

After reconstructing ˆ A1, ˆ A2, ..., ˆ A| ˆ Z| into a vector form ˆA1, ˆA2, ..., ˆA| ˆ Z| , we assume that

and calculate the expectation µs,ˆ z as well as standard deviation σs,ˆ z of ˆa1, ˆa2, ..., ˆaK for each observation s. Then, we use neural networks to learn independent skills ˆπθ(ˆa|s, ˆ z) by minimizing the KL divergence between ˆπθ(ˆa|s, ˆ z) and the empirical distribution ˆ p(ˆa|µs,ˆ z , σs,ˆ z ):

where we employ Gaussian distribution to characterize ˆπθ(ˆa|s, ˆ z) and ˆ p(ˆa|µs,ˆ z , σs,ˆ z). For more details about LIS, please refer to Algorithm 1.

5 Independent Skill Transfer (IST)

In this section, we propose to transfer independent skills to practical skills in a target environment by learning a transfer policy ϖφ([α, b]|s) as shown in Figure 4. Since primitive actions are linear combination of independent actions, the transfer policy ϖφ([α, b]|s) produces a weight αt and bias bt in each time step t based on observation st, so the composite action can be calculated by

where independent actions ˆa1 ∼ ˆπθ(ˆa|st,ˆ z1), ˆa2 ∼ ˆπθ(ˆa|st,ˆ z2), · · · , respectively, ⊗ denotes the kronecker product, and bt denotes the bias.

After executing the composite action at, the agent can achieve the observation st+1 ∼ T(st+1|st, at) and reward r(st, αt) from the target environment. Then, we employ SAC to update the transfer policy ϖφ([α, b]|s).

6 Experiment

In this section, we evaluate the behavior of independent skills in skill transfer 2 , where our experiments are conducted to answer the following three questions:

• Is our proposed collection of observation effective?
• Does our proposed IST outperform state-of-the-art?
• How is the performance of IST on difficult tasks?

In the pre-training stage, we employ DIAYN to learn 6 primitive skills (|Z| = 6), which can be used for both IST and primitive skill transfer (PST). According to the source environment HalfCheetah-v3 3 , we set up target environments by considering extra key elements as shown in Figure 5, including HalfCheetah-Hurdle (HCH), HalfCheetahAscending (HCA) and HalfCheetah-Upstair (HCU).

6.1 Collection of Observation

Figure 6 plots the loss of independent skills ˆπθ(ˆa|s, ˆ z) on training dataset and validation dataset. It is observed that the proposed strategy sampling from the key subset ˜ S performs better, where the loss converges to 10-8 on both training and validation sets, which indicates the validity of observation and action collection in characterizing the primitive skills π(a|s, z). Furthermore, as the trajectory length T increases from 150 to 250, a notable improvement can be obtained. Hence, we keep T = 250 in the following experiments. This answers the first question above.

6.2 Performance of IST

We compare IST with state-of-the-art to show the effectiveness and high efficiency of our proposed method.

2 https://github.com/qxtian/Learning-Independent-SKills

3 https://gym.openai.com/envs/

• (a) HCH: The agent requires to stride over a hurdle and walk to the destination, which becomes harder as d and h increase.

Figure 6: Convergence of loss, where the dark line and light color mark the loss on training set and validation set, respectively, and the sampling size from S (red) is identical with that of T =250 (black).

Compared with PST. Regarding PST [Peng et al., 2019], the composite action is thought of as the linear combination of primitive actions and SAC is adopted to learn the combined weights. While for IST, PCA is employed for skill actions, where the percentages 4 are 60.57%, 27.25%, 10.55%, 1.46%, 0.15% and 0.02%, respectively, as given in Figure 1. As seen, the last 3 eigenvalues are negligible. Hence, we consider to learn 3 independent skills (| ˆ Z| = 3) for the generation of practical skills. In such case, more than 98% component of primitive skills can be reserved in the conversion to independent skills. Furthermore, due to the independence between skills and lower dimension of weight α decreased from R|Z| to R| ˆ Z| , IST allows more efficient skill transfer. Consequently, as in Figure 7 (a)-(c), IST shows a higher efficiency of reward collection than PST for all cases. Particularly, from Figure 7 (f) where h = 0.3, the average return of IST converges in less than 150 episodes. In contrast, the average return of PST fails to converge, indicating the failure of skill transfer in finite episodes.

Compared with primitive skill selection (PSS). In PSS [Sharma et al., 2019], a network is trained to select an optimal skill from diverse pre-training primitive skills to deal with a specific task. In each step, a single primitive skill is selected and thus activated for transfer, but it is hardly as amenable as the combination of independent skills for practical skills. In fact, selecting a single skill can be a special case of linear combination of independent skills with weights αt = [0, · · · , 1, · · · , 0] and bias bt = 0. Therefore, it is observed from Figure 7 (a)-(c) that IST collects reward more efficiently and exhibits stronger generalization ability than PSS.

4 These percentages can be changed based on the real situation.

Compared with conventional RL. We compare the performance of IST with the existing reinforcement learning: SAC. Empirically, prior information of source environment has been reserved in primitive skills and then refined to independent skills as mentioned above. In contrast, SAC learning from scratch has no reutilization of primitive skills or prior environment. Hence, it is observed from Figure 7 (a)-(c) that, IST enables a higher learning efficiency than SAC learning from scratch by incorporating diverse prior experiences. Conclusively, the proposed method manages to reserve and refine sufficient prior information from the source environment for reutilization in a target environment.

6.3 Skill Transfer on Difficult Tasks

In this subsection, we evaluate the performance of skill transfer when the difficulty level gets increased. As shown in Table 1, when the given task gets harder (e.g. HCH h : 0.3→0.35 etc.), all skill transfer methods suffer from a degradation of performance in terms of success rate.

In such case, the proposed IST achieves the least degradation and the best performance compared with others (PST has nearly the same degradation, but less success rate), indicating that IST is less sensitive to the difficulty level of tasks. Furthermore, IST shows the best performance over most of tasks except for the negligible inferiority than SAC in HCA with u = 13.76◦ and 19.5◦ . Hence, the proposed IST exhibits its capability on difficult tasks, and we show in Figure 8 the frames of learned different tasks completed by IST.

7 Conclusion

In this work , we propose to learn independent skills from primitive skills and further transfer them to high-level complex tasks. Effective observation collection and independent skills guarantee the success of low-dimension skill transfer. Experiment results show a higher learning efficiency and stronger generalization ability of our proposed method.

Acknowledgements

We thank the funding support from the Westlake University and Bright Dream Joint Institute for Intelligent Robotics.