Dual Policy Distillation

1 Introduction

Reinforcement learning (RL), especially deep reinforcement learning has achieved great success in various domains [Sutton and Barto, 2018], ranging from robotic control [Levine et al., 2016], perfect information games [Silver et al., 2017] to imperfect information games [Zha et al., 2019a]. However, it usually requires a large number of interactions with the environment to obtain high-level performance [Salimans et al., 2017]. Recently, works have been proposed to study how we can transfer knowledge from one or more teacher models to a student model so that we can train an agent based on a pre-trained expert model [Rusu et al., 2016; Czarnecki et al., 2019]. One of the simple yet effective techniques is called policy distillation [Rusu et al., 2016], which uses supervised regression to train a student model to produce the same output distribution as the teacher model. Policy distillation has achieved great success and led to stronger performance in challenging domains [Teh et al., 2017; Yin and Pan, 2017]. Unfortunately, it is computationally expensive to obtain a teacher policy since a well-performed pre-trained model is often not available. In addition, the performance of the student model could be restrained by the teacher model if the teacher model is sub-optimal.

∗These two authors contributed equally in this work

In cognitive psychology, collaborative learning illustrates a situation in which a group of students works together to search for solutions [Dillenbourg, 1999]. Different from the traditional teacher-student relationship where students noninteractively receive information from teachers, collaborative learning involves joint intellectual efforts from diverse perspectives of students [Smith and MacGregor, 1992]. Motivated by this, we study the feasibility of collaborative learning on student policies without the use of pre-trained teacher models, and the methodology of extracting beneficial knowledge from a peer policy to accelerate learning, analogous to the human ability to learn from others.

In this paper, we introduce dual policy distillation (DPD), a student-student framework in which two policies operate on the same environment and extract knowledge from each other to benefit their learning. There are mainly two challenges in developing such dual learning framework upon contemporary learning-based RL agents. First, different from conventional policy distillation, which uses an expert policy, both policies in DPD are imperfect and may generate noisy outputs in the training process. It is unclear whether the regression to these noisy data is helpful. Second, contemporary RL algorithms usually use function approximators to approximate the policy function and the value function. The inaccurate estimations of the functions make it challenging to design practical algorithms that can be combined with learning-based agents.

To address the challenges above, apart from the original policy, we introduce another policy which is simultaneously trained in the same environment with different initialization. Each of the two policies iteratively optimizes its own RL

objective and updates a distillation objective which extracts knowledge from the other peer policy. One may find it surprising that the proposed framework tries to simultaneously encourage the uniqueness of the policy, and keep the two polices close with distillation objective. In the following sections, we demonstrate that this method is able to balance exploration and exploitation through parallelization and distillation, respectively, with two nice properties: (1) it does not require an expert policy as teacher signals in the sense that the two student policies explore different aspects of the environment and share knowledge with each other; (2) distilling knowledge from a peer policy has theoretical policy improvement and it can achieve satisfactory performance with a learning-based agent and function approximation in our empirical results.

Through addressing the challenges, in this paper, we make the following contributions:

• We introduce dual policy distillation (DPD), a studentstudent framework in which two polices extract beneficial knowledge from each other to help their learning.
• We provide a theoretical justification of the policy improvement of DPD. We show that in the ideal case, by distilling a hypothetical hybrid policy, each of the policies has guaranteed policy improvement.
• We propose a practical algorithm 1 based on our theoretical results. The algorithm uses a disadvantageous policy distillation strategy which prioritizes the distillation at disadvantage states and pushes each of the two policies towards the optimal policy. Experiments on several continuous control tasks demonstrate that the proposed DPD significantly enhances each of the two policies.

2 Preliminaries

In this section, we introduce reinforcement learning (RL), the background of policy distillation, and the notations used in this paper.

In the following, we consider standard reinforcement learning which is denoted by a sextuple (S,A,PT ,R, γ, p0), where S is the set of states, A is the set of actions, PT : S×A → S is the state transition function, R : S×A×S → R is the reward function, γ ∈ (0, 1) is the discount factor, and p0 is the distribution of the initial state. The interactions with the environment can be formalized as a Markov decision process: at each timestep t, an agent takes an action at ∈ A at state st ∈ S and observes the next state st+1 with a reward signal rt. This results in a trajectory τ which consists of a sequence of triplets of states, actions and rewards, i.e., τ = {(st, at, rt)}t=1,...,T , where T is the terminal timestep. The objective of RL algorithm is to learn a policy π : S → A that maximizes the cumulative reward R = E[∑∞t=1 γ t rt].

We use standard definitions of value function, stateaction value function, and advantage function of a policy π, i.e., V π (st) = Eat,st+1,...|π[∑∞0 γ l rt+1], Qπ (st, at) = Est+1,at+1,...|π[∑∞t=0 γ l rt+1], and Aπ (st, at) = Qπ (st, at)- V π (st), where at, st+1, ...|π and st+1, at+1, ...|π denote the resulting trajectories from the environment if we follow policy

1 https://github.com/datamllab/dual-policy-distillation

• π. We use ρπ(s) to denote discounted visitation frequencies of state s, that is , ρπ(s) = ∑∞t=0 γ t p(st = s), where p(st = s) is the probability of s being visited at timestep t.

Policy distillation is a simple yet effective method of transferring knowledge from one or more action policies to an untrained network. We denote ˜ π as a teacher policy, i.e., a trained model that can generate expert data, and πθ as an untrained parametric student policy. Policy distillation trains the student policy by conducting regression to the teacher policy, i.e., minimizing the following objective:

where s ∼ ˜ π means s follows the distribution of ρπ(·), and D(·, ·) is a kind of distance metric. The above description is a general form of policy distillation. There are multiple choices for the distance metrics, such as mean square error, KL divergence or log-likelihood loss [Rusu et al., 2016]. In this paper, we use mean square error for its simplicity.

Note that, in our presented algorithm, we use ˜ πφ to denote a trainable peer policy with parameters φ. πθ and ˜ πφ are both student models and are trained interactively.

3 Dual Policy Distillation

In this section, we present dual policy distillation (DPD), a framework that enables knowledge transfer between two student policies operating on the same environment. We consider two policies denoted as π and ˜ π respectively. We first theoretically justify that extracting knowledge from a peer policy will lead to policy improvement through a view of hypothetical hybrid policy. Then based on our theoretical results, we present a disadvantageous policy distillation objective that can be combined with learning-based RL algorithms. Figure 1 shows an overview of the proposed framework.

3.1 AView of Hypothetical Hybrid Policy

We first justify that π and ˜ π are theoretically complementary, and thus transferring knowledge between two policies will lead to policy improvement for both π and ˜ π. Consider a hypothetical hybrid policy:

where ξ ˜ π (s) = V ˜ π (s) - V π (s) represents the advantage of ˜ π over π at state s. That is, the hypothetical policy πhypo selectively follows one of π and ˜ π depending on which policy has larger expected discounted reward at the state. In Proposition 1, we give that in ideal case the hypothetical hybrid policy is an improved policy compared to π or ˜ π.

Proposition 1. For the hypothetical hybrid policy πhypo defined in Eq. 2, we have ∀s ∈ S, V πhypo (s) ≥ V π (s) and V πhypo (s) ≥ V ˜ π (s).

Proof: Considering a state s ∈ S, we prove that V πhypo (s) ≥ V π (s). Define advantage policy at s:

That is, πadv s is one of π and ˜ π such that it has higher value at state s. Note that whether πadv s (·|s′ ) is π or ˜ π depends on s instead of s′ , where s′ is a different state. It is straightforward to have

where s′ is another state and s ̸= s′ .

We now consider an arbitrary state si ∈ S and denote si+1 as the next state. Define

where n is a positive integer. That is, the value in state si if we follow πhypo for the first n steps and follow πadv si+n afterwards.

When n = 0 we have V0(si) = V πadv si (si). We first prove ∀si ∈ S, V1(si) ≥ V0(si).

Based on the equations above, when k ≥ 1, given that

∀si ∈ S, Vk(si) ≥ Vk-1(si), we have

By induction, we can conclude that ∀n ≥ 0, ∀s ∈ S, Vn(s) ≥ V πadv s (s) ≥ V π (s). Thus, ∀s ∈ S, V πhypo (s) ≥ V π (s). Similarly, we have ∀s ∈ S, V πhypo (s) ≥ V ˜ π (s). The above proof is also applicable in continuous space if we replace sum operation with integration.

Directly following the well-known policy improvement theorem [Sutton and Barto, 2018], Proposition 1 suggests that, the hypothetical hybrid policy πhypo defined in Eq. 2 is at least as good as π and ˜ π. If V π (s) > V ˜ π (s) at some states and V π (s′ ) < V ˜ π (s′ ) at some other states, πhypo will be strictly better than π and ˜ π. Our empirical observation also supports this intuition (see Figure 3). Thus, it will lead to theoretical policy improvement if we let π and ˜ π conduct regression to the hypothetical hybrid policy πhypo .

3.2 Disadvantageous Policy Distillation

Based on the above theoretical results, we introduce a practical dual distillation strategy which can be combined with learningbased RL agents.

For a practical algorithm, rather than building the hypothetical hybrid policy πhypo at every step, we prefer to use an objective to train the policy. In Proposition 2, we give that under mild conditions, the distillation of πhypo is equivalent to minimizing a simple objective.

Proposition 2. The distillation to π from the hypothetical hybrid policy πhypo defined in Eq. 2 is equivalent to minimizing the following objective:

where 1(·) is the indicator function and D(·, ·) is denoted as the distance metric.

Proof: Since the two policies π and ˜ π have similar state visiting frequency, the distillation to πhypo defined in Eq. 1 can be rewritten as follow:

Algorithm 1 DPD: dual policy distillation

• 1: Input: policy πθ, peer policy ˜ πφ, and maximum iteration number M
• 2: for iteration = 1,M do
• 3: Execute πθ to generate trajectories and save them into buffer Bπ
• 4: Update πθ based on its RL objective
• 5: Update πθ based on Eq. 11
• 6: Execute ˜ πφ to generate trajectories and save them into buffer B˜ π
• 7: Update ˜ πφ based on its RL objective
• 8: Update ˜ πφ based on Eq. 12
• 9: end for

The difference between ρπ(·) and ρ˜ π(·) can be ignored in our dual learning setting, since the dual distillation will push the two policies to perform similar actions and hence will result in similar state visiting frequencies. Our empirical observations also support this assumption in that both the learning curves and the outputs of the policy network are very similar during the learning process (see Figure 3). The result of Eq. 9 suggests a simple intuition: the states at which the peer policy is advantageous will be more helpful in distillation; otherwise, maintaining the current policy at the state would be a better choice. We call this strategy disadvantageous policy distillation because it prioritizes the distillation of the states at which the current policy is more disadvantageous than the peer policy.

For now, we ignore the estimation error for the value functions which are usually approximated by deep neural networks. However, in the approximate setting, we have to consider the inaccurate estimations of value functions, which makes it difficult to optimize Eq. 9. In our preliminary experiments, we observed that directly update the policies based on Eq. 9 will misclassify many states and lead to sub-optimal performances. Therefore, we propose to soften Eq. 9 and introduce weighted objectives J w πθ (θ) and J w ˜ πφ (φ) to update parametric policies πθ and ˜ πφ:

where exp(αξ ˜ πφ (s)) and exp(αξ πθ (s)) are confidence scores, and α controls the confidence level which should be chosen depending on how accurate the value function estimation is.

We now describe how we combine the objectives in Eq. 11 and 12 with a learning-based agent. Given two parametric policies πθ and ˜ πφ operating on the same environment, we use an alternating update framework as follows. In the first step, policy πθ is updated based on its own RL objective. In the second step, πθ is updated based on the distillation objective. Specifically, we sample a mini-batch of transitions from the buffer of ˜ πφ and compute the advantage ξ ˜ πφ (s) for each sampled state and update the policy based on the distillation objective. We then do the same updates to its peer policy ˜ πφ. As a result, each policy learns to optimize its RL

objective and simultaneously extracts useful knowledge from its peer policy to enhance itself. The algorithm is summarized in Algorithm 1.

3.3 Connection to Value Iteration

In this subsection, we justify that the proposed distillation objective will lead to similar effects as by classical value iteration [Sutton and Barto, 2018]. Let π∗ be an optimal policy and V π∗ (s) be the optimal values, i.e., the expected discounted future rewards if we start from s and follow the optimal policy. Value iteration method iteratively updates the state values as follow:

where Vi(·) and Vi+1(·) are the values of the current step and the next step respectively. The intuition of this update rule is to compute the maximum value by choosing the most valuable action in each iteration to push the values towards and finally converge to the optimal values. Define S˜ π as the set of disadvantage states, i.e, S˜ π = {s|V ˜ π (s) > V π (s)}. Then it is straightforward to have

The distillation of the states in S˜ π has similar effects for π. Encouraging π to choose the actions from ˜ π for the states in S˜ π will push the values of π towards optimal values because these actions lead to larger values based on V ˜ π (·). Thus, both Eq. 11 and 12 will optimize the values of the two policies.

4 Experiments

In this section, we empirically evaluate the proposed dual policy distillation (DPD) framework. We develop two instances of DPD by two benchmark RL algorithms, and evaluate them on four continuous control tasks. Our experiments are designed to answer the following questions:

• Q1: Is DPD able to improve the performance in both onpolicy and off-policy settings (Sec. 4.2)?
• Q2: How will the values and actions outputted by the models evolve during training (Sec. 4.3)?

4.1 Experimental Setting

Our experiments are implemented upon PPO [Schulman et al., 2017] and DDPG [Lillicrap et al., 2016], which are benchmark RL algorithms implemented in OpenAI baselines 2 . We follow all the hyper-parameters setting and network structures for our DPD implementation and all the baselines we considered. Since the policy network in DDPG is deterministic, we directly use the outputs of the Q-network in DDPG to estimate the state values. We use mean square error to compute the distance between the output actions of the two policies for the distillation objective. We consider the following baselines:

• DDPG: the vanilla DDPG without policy distillation.
• PPO: the vanilla PPO without policy distillation.

2 https://github.com/openai/baselines

The experiments are conducted on several continuous control tasks from OpenAI gym 3 [Brockman et al., 2016]: Swimmer-v2, HalfCheetah-v2, Walker2d-v2, Humanoid-v2.

4.2 Overall Performance

We conduct experiments on both on- and off-policy settings to validate the proposed disadvantageous distillation strategy is beneficial in general. Since DPD requires updating two policies simultaneously, we run vanilla DDPG and PPO for twice the number of timesteps as that of DPD for a fair comparison, i.e., we run DPD-DDPG and DPD-PPO for 2.5 × 106 and 1 × 107 timesteps; and vanilla DDPG and PPO for 5 × 106 and 2 × 107 timesteps. Figure 2 plot the learning curve of offpolicy settings, and the result of on-policy settings is tabulated in Table 1.

Based on the result, we observed that, each of the two policies in DPD is significantly enhanced by the dual distillation, and DPD outperforms vanilla DDPG and PPO within the same running timesteps in the 4 tasks. Specifically, when comparing DPD and DDPG at 2.5 × 106 timesteps, the maximum return of DPD has an improvement of more than 15 percent in 3 out of 4 tasks. By comparing the performance of DPD at 2.5×106 timesteps with that of both PPO and DDPG at 5×106 timesteps, the maximum return of DPD has an improvement of more than 10 percent in the tasks. In our experiments, we only empirically explore α over a small set. It is possible to further improve DPD if exploring more α values. The results suggest that it is promising to exploit the knowledge from a peer policy and the proposed framework is generally applicable for both on- and off-policy algorithms.

4.3 Analysis of the Dual Distillation

We study how the Q-values and the actions outputted by the two learners evolve under DPD framework. We randomly sample some states from the rollouts performed by a pretrained model which is obtained by training DDPG for 5×106 timesteps. We then feed these states to the DPD and DDPG models in different training stages. The outputted Q-values and actions are illustrated in Figure 3.

3 https://gym.openai.com/

For the Q-values, we make two observations. First, in all the training stages, the Q-values outputted by each of the learners tend to be larger at some states and smaller at some other states compared with the other learner. The result supports our hypothesis that the two learners are complimentary so that we can find a hypothetical hybrid policy that has guaranteed policy improvement. Second, we can see that the Q-values of all the states tend to increase throughout the training process, and they increase faster than those of DDPG. The result suggests that the proposed dual distillation indeed pushes the values of the two learners towards the larger optimal values, as indicated by our theoretical justification.

For the actions, we calculate the Euclidean distance between the outputted actions of the two learners in DPD, and run two separate DDPG models, calculate the Euclidean distance between their outputted actions for comparison. We observe that the two learners in DPD tend to perform more similar actions than the two DDPG models. The average Euclidean distance for DPD decreases from the early stage (0.43) to the later stage (0.31), which suggests that the dual distillation may make the two learners slowly converge through the training.

Since the two learners have similar policies, the result further supports our assumption that the two learners in DPD have similar state visiting frequencies.

5 RelatedWork

5.1 Teacher-student Framework

A typical teacher-student framework is imitation learning. [Abbeel and Ng, ; Finn et al., 2016]. One representative method is behavior cloning [Bain and Sommut, 1999; Ho and Ermon, 2016; Rusu et al., 2016], which directly trains a student policy based on demonstration data of experts. Previous work shows that it is promising to learn from imperfect demonstrations [Hester et al., 2018] and exploiting own past good experiences will help exploration [Oh et al., 2018]. Our framework also learns from imperfect demonstrations, but treats actions performed by a peer policy as demonstrations. Meta-learning methods is studied to make use of a teacher model to improve the sample efficiency [Xu et al., 2018b; Xu et al., 2018a; Zha et al., 2019b]. Our work extends the traditional teacher-student setting and studies a student-student framework with two student models distilling knowledge from each other. Each model servers as both student and teacher and work together with its peer model to find the solution.

5.2 Multi-agent Reinforcement Learning

Multi-agent reinforcement learning [Littman, 1994; Tan, 1993; Shoham et al., 2003] studies how a group of agents sharing the same environment learns to collaborate and coordinate with each other to achieve a goal. There are several studies on knowledge diffusion in multi-agent setting [Hadfield-Menell et al., 2016; Da Silva et al., 2017; Omidshafiei et al., 2019]. Although our work also introduces two learners, our setting is significantly different from the multi-agent setting in that the two learners in the framework are independently deployed to two instances of the same single-agent environment. The two learners capture various aspects of the same environment and share beneficial knowledge during the training process.

6 Conclusion and FutureWork

In this work, we introduce dual policy distillation, a studentstudent framework which enables two policies to explore different aspects of the environment and exploit the knowledge from each other. Specifically, we propose a distillation strategy which prioritizes the distillation at disadvantage states from the peer policy. The theoretical and empirical results show the promising of the proposed framework. In the future, we will investigate whether our framework can be extended to enable knowledge transfer among multiple tasks and explore the possibility of using our framework to combine the benefits of different RL algorithms.

Acknowledgements

This work is, in part, supported by NSF (#IIS-1750074, #IIS1939716 and #IIS-1900990). The views and conclusions contained in this paper are those of the authors and should not be interpreted as representing any funding agencies.