Boosting Offline Reinforcement Learning with Residual Generative Modeling

1 Introduction

Reinforcement learning (RL) has achieved promising results in many domains [Mnih et al., 2015; Silver et al., 2017; Kalashnikov et al., 2018; Berner et al., 2019; Vinyals et al., 2019; Ye et al., 2020b; Ye et al., 2020a]. However, being fettered by the online and trial-and-error nature, applying RL in real-world cases is difficult [Dulac-Arnold et al., 2019]. Unlike supervised learning which directly benefits from large offline datasets like ImageNet [Deng et al., 2009], current RL has not made full use of offline data [Levine et al., 2020]. In many real-world scenarios, online exploration of the environment may be unsafe or expensive. For example, in recommendation [Li et al., 2011] or healthcare domains [Gottesman et al., 2019], a new policy may only be deployed at a low frequency after extensive testing. In these cases, the offline dataset is often large, potentially consisting of years of logged experiences. Even in applications where online exploration is feasible, offline data is still beneficial. For example, in strategy video games (Dota or Honor of Kings) that require highly complex action control, current RL methods still learn from the scratch, resulting in a long time to master human-level skills [Berner et al., 2019; Ye et al., 2020c], where there is a large amount of logged replay data from vast players to be utilized.

Although off-policy RL methods [Mnih et al., 2015; Lillicrap et al., 2015; Haarnoja et al., 2018] may be executed in the offline scenario directly, their performance was shown to be poor [Fujimoto et al., 2019] with a fixed batch of data and no interactions with the environment. The poor performance is suspected due to incorrect value estimation of actions outside the training data distribution [Kumar et al., 2019]. The error of estimated values will accumulate and amplify during the Bellman backup process in RL. Typical off-policy RL would improve the learned policy by trying out the policy in the environment to correct the erroneous estimations, which is not applicable in purely offline scenarios.

Instead of correcting erroneous estimations by interacting with the environment, offline RL provides an alternative for the offline scenario. Current representative offline RL methods [Fujimoto et al., 2019; Kumar et al., 2019; Wu et al., 2019; Kumar et al., 2020] mainly study how to reduce the error with conservative estimations, i.e., constraining the action or state-action distribution around the given dataset's distribution when learning value functions. With such estimations, the learned RL policy can approach or exceed the original policy (also called behavioral policy) that generates the fixed dataset. As most offline RL methods assume that the behavioral policy is unknown, typical offline RL methods would first approximate the behavioral policy through generative modeling [Levine et al., 2020], i.e., learn to output actions for given states. Then, the objective is to learn approximations for selecting the highest valued actions that are similar to the approximated behavioral policy.

Different from the aforementioned methods, this paper focuses on investigating the impact of errors from generative modeling on offline RL. As we will show, in value-based offpolicy scenarios, the error of generative modeling will accumulate in the process of learning the Q-value function during Bellman backup. Our main contribution involves studying the accumulating process of generative modeling error in offline RL and developing a practical method to mitigate this error. To expand, we first examine the error accumulation process during offline RL learning in detail and analyze the influence of generative modeling error on the final offline RL error on the Q-value function. Then we propose an error reduction method via residual learning [Huang et al., 2017]. Through experiments on a set of benchmark datasets, we verify the effectiveness of our method in boosting offline RL performance over state-of-the-art methods. Furthermore, in the scenario of the multiplayer online battle arena (MOBA) game Honor of Kings, which involves large state-action space, our proposed method can also achieve excellent performance.

2 RelatedWork

Under fully offline settings where no additional online data collection is performed, both offline RL methods and imitation learning methods can be used to learn a policy from pre-recorded trajectories.

Offline RL. Offline RL describes the setting in which a learner has access to only a fixed dataset of experience, in contrast to online RL which allows interactions with the environment. Existing offline RL methods suffer from issues pertaining to OOD actions [Fujimoto et al., 2019; Kumar et al., 2019; Wu et al., 2019; Kumar et al., 2020]. Prior works aim to make conservative value function estimations around the given dataset's distribution, and then only use action sampled from this constrained policy in Bellman backups for applying value penalty. Different from these works, this paper focuses on how the errors propagate throughout the whole process of offline RL, from generative modeling to value approximation.

Imitation learning. Imitation learning (IL) is to learn behavior policies from demonstration data [Schaal, 1999; Hussein et al., 2017]. Though effective, these methods are not suitable for offline RL setting because they require either onpolicy data collection or oracle policy. Different from offline RL, imitation learning methods do not necessarily consider modeling the long-term values of actions or states like reinforcement learning methods.

3 Preliminaries

3.1 Reinforcement Learning

We consider the environment as a Markov Decision Process (MDP) (S,A, P,R, γ), where S is the state space, A is the action space, P : S × A × S → [0, 1] denotes the state transition probability, R : S × A → R represents the reward function, and γ ∈ (0, 1] is the discount factor. A policy π is a mapping S × A → [0, 1].

A value function provides an estimate of the expected cumulative reward that will be obtained by following some policy π(at|st) when starting from a state-action tuple (st, at) in the case of the state-action value function Qπ (st, at):

In the classic off-policy setting, the learning of Q-function is based on the agent's replay buffer D that gathers the experience of the agent in the form of (st, at, rt, st+1), and each new policy πk collects additional data by exploring the environment. Then D, which consists of the samples from π0, . . . , πk, is used to train a new updated policy πk+1.

3.2 Offline Reinforcement Learning

Offline RL additionally considers the problem of learning policy π from a fixed dataset D consisting of single-step transitions (st, at, rt, st+1), without interactions with the environment. This is in contrast to many off-policy RL algorithms that assume further interactions with the current policy rolling out in the environment. In this paper, we define the behavioral policy πb as the conditional distribution p(a|s) in the dataset D, which is treated unknown. In real-world cases, the training dataset could be generated by a collection of policies. For simplicity, we refer to them as a single behavioral policy πb.

Value Function Approximation

For large or continuous state and action spaces, the value function can be approximated with neural networks ˆQθ, parameterized by θ. With the notion of Bellman operator T π , we can denote Equation (1) as ˆQπ = T π ˆQπ with γ ∈ [0, 1). This Bellman operator has a unique fixed point that corresponds to the true state-value function for π(a|s), which can be obtained by repeating the iteration ˆQπk+1 = T π k ˆQπ , and it can be shown that limk→∞ ˆQπk = T π ˆQπ .

Offline RL algorithms based on this value function approximation with the iterative update are shown to suffer from action distribution shift [Fujimoto et al., 2019] during training. Since the policy in next iterate is computed by choosing actions that greedily maximize the state-action value function at each state, πk+1(a|s) = argmaxa Qπk (s, a), it may be biased towards out-ofdistribution (OOD) actions with erroneously high Q-values. In RL with explorations, such errors can be corrected by rolling out the action in the environment and observing its true value. In contrast, an offline RL agent is unable to query such information from the environment. To mitigate the problem caused by OOD actions, typical offline RL methods focus on constraining the learned policy to output actions that lie in the training distribution πb .

Generative Modeling

Because we do not assume direct access to πb, it is common in previous work to approximate this behavior policy via a generative model Gω(s), with max-likelihood over D [Fujimoto et al., 2019; Kumar et al., 2019; Wu et al., 2019; Levine et al., 2020]:

We denote the approximated policy as ˆ πb and refer to it as "cloned policy" to distinguish it from πb.

In Section 4, we will analyze how the errors propagate from generative modeling to value function approximation, resulting in the overall errors for offline RL. Then we will introduce how to reduce the overall errors of offline RL by reducing the generative modeling error, with theoretical proofs and implemented models in Section 5.

4 Generative Modeling Error Propagation

In this section, we define and analyze how the generative modeling error propagates during the process of value estimation in offline RL. We derive bounds which depend on the generative modeling. This motivates further focusing on mitigating generative modeling error.

4.1 Generative Modeling Error

As discussed in the last section, we need to approximate πD(s, a) with a generative model Gω(s). If we train Gω(s) with supervised learning (i.e., standard likelihood maximization) on D, we have the following result from [Ross et al., 2011], whose proof can be found in [Levine et al., 2020].

Lemma 1 (Behavioral cloning error bound). If τ πb (s) is the state distribution induced by πb and π(a|s) is trained via standard likelihood maximization on s ∼ τ πb (s) and optimal labels a, and attains generalization error ϵg on s ∼ τ πb (s), then l(π) ≤ C + H2 ϵg is the best possible bound on the expected error of the learned policy, where C is the true accumulated reward of πb .

This means that even with optimal action labels, we still get an error bound at least quadratic in the time horizon H in the offline case. Intuitively, the policy ˆπb learned with generative model Gω(s) may enter into states that are far outside of the training distribution, where the generalization error bound ϵg no longer holds on unseen states during training. Once the policy enters one OOD state, it will keep making mistakes and remain OOD for the remainder of the testing phase, accumulating O(H) error. Since there is a non-trivial chance of entering an OOD state at every one of the H steps, the overall error scales as O(H2 ).

4.2 Error Propagation on Value Estimation

Definition 1 (Value function approximation error). We define ϵπD (s, a) as the value function approximation error between the true state-action value function QπD computed from the dataset D and the true state-action value function Q∗ :

Theorem 1. Given a policy πb that generates the dataset D, if we model its cloned policy ˆπb from D with a generative modeling error of ϵg, assume that δ(s, a) = supD ϵπD (s, a) and η(s, a) = supD ϵg(a|s), with the action space of dimension |A|, δ(s, a) satisfies the following:

Proof. Firstly, we have the following:

The proof follows by expanding each Q, rearranging terms, simplifying the expression and then representing cloned policy π with behavior policy πb with a generative error ϵg.

Based on above equation, we take the supremum of ϵπD (s, a) and have the following:

For a finite, deterministic MDP, if all possible state transitions are captured in D, pD(s′ |s, a) will be equivalent to p∗ (s′ |s, a), we will have ϵπD (s, a) = 0. However, in infinite or stochastic MDP, it might require an infinite number of samples to cover the true distribution. Therefore, δ(s, a) = 0 if and only if the following strong assumptions holds: the true MDP is finite and deterministic, and all possible state transitions are captured in D. Otherwise, we have δ(s, a) > 0.

From Theorem 1, we have δ(s, a) scales as O(η|A|), where η(s, a) = supD ϵg(a|s). Intuitively, this means we can prevent an increase in the state-action value function error by learning a generative model Gω(s) with smaller ϵg. Meanwhile, for settings where the action space is small, the ϵg are will have smaller influences in inducing the state-action value function error. Overall, in the time horizon H, since the generative error ϵg scales as O(H2 ), δ(s, a) = supD ϵπD (s, a) scales as O(|A|H2 ).

5 Residual Generative Modeling

We begin by analyzing the theoretical properties of residual generative modeling in a deterministic policy setting, where we are able to measure the monotonically decreasing loss over sampled actions precisely. We then introduce our deep reinforcement learning model in detail, by drawing inspirations from the deterministic analog.

5.1 Addressing Generative Modeling Error with Residual Learning

In this section, we will provide a short theoretical illustration of how residual learning can be used in addressing the generative modeling under deterministic policies. In this example, we will assume that ρφ(s) is the original generative model in mimicking the policy πb in dataset D, and the residual generative model is denoted as ˆa = Gω(s, ρφ(s)), where ω stands for the parameters used to combine input s and generative model ρφ(s).

Without loss of generality, we can re-write the final output layer by assuming that Gω is parameterized by a linear weight vector w and a weight matrix M, and previous layers can be represented as Gω2 (s). Thus, ˆa can be denoted as ˆa = wT (MGω2 (s) + ρφ(s)). We have the following result from [Shamir, 2018]:

Lemma 2. Suppose we have a function defined as

where l is the defined loss, a, B are weight vector and matrix respectively, and ψ is the parameters of a neural network. Then, every local minimum of Γ satisfies

if following conditions are satisfied: (1) loss l(ˆ y, y) is twice differentiable and convex in ˆ y; (2) Γψ(a,B), ▽Γψ(a,B), and ▽2Γψ(a, B) are Lipschitz continuous in (a, B).

Theorem 2. When using log loss or squared loss for deterministic policy and linear or convolution layers for Fω2 , every local optimum of ˆa = wT (MGω2 (s) + ρφ(s)) will be no worse than ˆa = wT ρφ(s) .

Proof. For deterministic policies, Gω2 (x) is Lipschitz continuous when using linear or convolution layers [Virmaux and Scaman, 2018]. Since log loss and squared loss are twice differentiable in a, we have

where Γψ(a,B) = Γ(a,B, ψ) .= Ex,y[l(aT (H(x) + BFθ(x)), y)], l is the defined loss, a , B are weight vector and matrix respectively, and θ is the parameters of a neural network.

That is, the action-conditioned model has no spurious local minimum that is above that of the original generative model ρφ(s).

5.2 Residual Generative Model

The main difference between our method and existing offline RL methods is that we design a residual modeling part for the generative model when approximating the πD. Therefore, in this section, we mainly introduce our approach to offline reinforcement learning, AQL (action-conditioned Q-learning), which uses action-conditioned residual modeling to reduce the generative modeling error.

Our generative model consists of two major components: a conditional variational auto-encoder (VAE) that models the distribution by transforming an underlying latent space, and a residual neural network that models the state-action distribution residuals on the output of the VAE.

Conditional VAE

To model the generative process of predicting actions given certain states, analogous to existing literatures like BatchConstrained Q-learning (BCQ) [Fujimoto et al., 2019] and Bootstrapping error reduction (BEAR) [Kumar et al., 2019], we use a conditional VAE that takes state and action as input and outputs the reconstructed action. Given the raw state, we first embed the state observation with a state embedding module in conditional VAE. Then in the encoder part, we concatenate the state embedding with action input and output the distribution of latent space (assumed to be Gaussian for continuous action space and Categorical for discrete action space). In the decoder part, we concatenate the state embedding and the latent variable z from the learned distribution and outputs the reconstructed action. The overall training loss for the conditional VAE is:

where the first term is the reconstructed loss, and the second term is the regularizer that constrains the latent space distribution, s is the state input to the conditional VAE, a is the action in the dataset in pair with s.

Residual Network

Unlike BCQ and BEAR which take VAE or a single feedforward neural network as the generative model, we propose to use the reconstructed action from VAE as an additional input to learn the residual of action output. This residual mechanism is motivated to boost offline RL by reducing the generative model's error. The overall loss for the generative network is:

where ˆ a = Gω(s) is the output of the residual network, and d(ˆ a, a) is the distance measure between two actions. For continuous actions, d could be defined as the mean squared error; for discrete actions, d could be defined as the cross-entropy. Intuitively, this loss function includes the original generative modeling loss function (usually treated the same as a behavior cloning loss) and a VAE loss, optimized at the same time.

5.3 Training Process

We now describe the practical offline RL method based on BEAR, a similar variant on BCQ or other methods can be similarly derived. Empirically we find that our method based on BEAR performs better. We've described our generative model in previous sections, here we briefly introduce the other part of the offline RL algorithm, i.e., value function approximation process, which is similar to BCQ and BEAR. To compute the target Q-value for policy improvement, we use the weighted combination of the maximum and the minimum Q-value of K state-action value functions, which is also adopted in BCQ and BEAR and shown to be useful to penalize uncertainty over future states in existing literature [Fujimoto et al., 2019]:

where s′ is the next state of current state s after taking action a, θ′j is the parameters for target network, γ is the discount factor in Bellman Equation, λ is the weighting factor.

Following BEAR, we define the generative modeling update process as a constrained optimization problem, which tries to improve the generator and constrains the policy within a threshold ϵ:

where α is the tradeoff factor that balances constraining the policy to the dataset and optimizing the policy value, and can be automatically tuned via Lagrangian dual gradient descent for continuous control and is fixed as a constant for discrete control.

This forms our proposed offline RL method, which consists of three main parameterized components: a generative model Gω(s), Q-ensemble {Qθi }Ki=1 , target Q-networks {Qθ′i }Ki=1 and target generative model Gω′ . In the following section, we demonstrate AQL results in mitigating the generative modeling error and a strong performance in the offline RL setting.

6 Experiment

We compare AQL to behavior cloning, off-policy methods and prior offline RL methods on a range of dataset compositions generated by (1) completely random behavior policy, (2) partially trained, medium scoring policy, and (3) an optimal policy. Following [Fujimoto et al., 2019; Kumar et al., 2019; Wu et al., 2019], we evaluate performance on four Mujoco [Todorov et al., 2012] continuous control environments in OpenAI Gym [Brockman et al., 2016]: HalfCheetah-v2, Hopper-v2, and Walker2d-v2. We evaluate offline RL algorithms by training on these fixed datasets provided by openaccess benchmarking dataset D4RL [Fu et al., 2020] and evaluating the learned policies on the real environments. The statistics for these datasets can be found in [Fu et al., 2020].

Baselines. We compare with off-policy RLmethods - Deep Deterministic Policy Gradient (DDPG) [Lillicrap et al., 2015] and Soft Actor-Critic (SAC) [Haarnoja et al., 2018]), Behavior Cloning (BC) [Ross and Bagnell, 2010], and stateof-the-art off-policy RL methods, including BCQ [Fujimoto et al., 2019], BEAR [Kumar et al., 2019], Behavior Regularized Actor Critic with policy (BRAC-p) or value (BRACv) regularization [Wu et al., 2019], and Conservative QLearning (CQL(H)) [Kumar et al., 2020]. We use the opensource codes provided in corresponding methods and keep the same parameters in our proposed method for a fair comparison.

Experimental settings. To keep the same parameter settings as BCQ and BEAR, we set K = 2 (number of candidate Q-functions), λ = 0.75 (minimum weighting factor), ϵ = 0.05 (policy constraint threshold), and B = 1000 (total training steps). We report the average evaluation return over three seeds of the learned algorithm's policy, in the form of a learning curve as a function of the number of gradient steps taken by the algorithm. The samples collected during the evaluation process are only used for testing and not used for training.

6.1 Results

Effectiveness of mitigating generativemodeling error. In previous sections, we argued in favor of using the residual generative modeling to decrease the generative modeling error. Revisiting the argument, in this section, we investigate the empirical results on the error between true a at state s from D and the generated action ˆa from Gω(s). In BCQ, it uses a vanilla conditional VAE as Gω(s); BEAR use a simple feed-forward neural network. Our proposed method combines these two models with a residual network.

• Figure 1 shows the comparison of the error, with the mean and standard deviation from the last 500 training batches of each method. We have the following observations:
• Both AQL and BEAR have a lower error in generative modeling than BCQ, and both AQL and BEAR keep the errors below 0.05 in most cases. This is because they use the dual gradient descent to keep the target policy constrained below the threshold ϵ that is set as 0.05, while BCQ does not have this constraint.
• Our proposed method AQL has a consistent lower error than BEAR. This is because AQL uses residual learning to mitigate the generative modeling error, which matches our analysis, as suggested by Theorem 2.

Effectiveness of boosting offline RL. As analyzed in previous sections, we can prevent an increase in the state-action value function error by learning a generative model Gω(s) with smaller ϵg, and thus learn a better offline RL policy. Here, we investigate the effectiveness of the proposed method on boosting the performance of offline RL.

Table 1 shows the comparison of our proposed method over behavior cloning, off-policy methods, and state-of-the-art offline RL methods. We have the following observations:

• Off-policy methods (DDPG and SAC) training in an purely offline setting yields a bad performance in all cases. This is due to the incorrect estimation of the value of OOD actions, which matches the existing literature.
• Offline RL methods can outperform BC under datasets generated by the random and medium policy. This is because BC simply mimicking the policy behavior from the dataset without the guidance of state-action value. Since the dataset D is generated by a non-optimal policy, the policy learned by BC could generate non-optimal actions. This non-optimality could accumulate as the policy rolls out in the environments.
• AQL performs similarly or better than the best prior methods in most scenarios. We noticed that BRAC and CQL(H) yields better performance in random and medium data than in expert data. This is because under expert data, the variance of the state-action distribution in the dataset might be small, and simply mimicking the behavior policy could yield satisfactory performance (like BC). While BRAC and CQL(H) does not have any specific design to reduce the generative error, AQL has a better generative model to mimicking the distribution of D and thus consistently performs as one of the best methods cross most settings. As analyzed in previous sections, we can lower the error of state-action value estimation, thus boost the offline RL performance.

We also noticed that although the overall performance of AQL is slightly better than CQL(H), since CQL(H) has additional constraints that AQL does not consider. We added the constraints of CQL(H) into AQL and found out that AQL can further improve on CQL(H) in all cases, which means the effectiveness of boosting offline RL methods by reducing the generative error.

6.2 Case Study: Honor of Kings 1v1

An open research problem for existing offline RL methods is the lack of evaluations on complex control tasks [Levine et al., 2020; Wu et al., 2019] with large action spaces. Therefore, in this paper, we further implement our experiment into Table 1: Performance of AQL and prior methods on gym domains from D4RL, on the unnormalized return metric, averaged over three random seeds, with top-3 emphasized. While BCQ, BEAR, BRAC and CQL(H) perform unstably across different scenarios, AQL consistently performs similarly or better than the best prior methods.

a multiplayer online battle arena (MOBA) game, the 1v1 version of Honor of Kings (the most popular and widely studied MOBA game at present [Ye et al., 2020a; Ye et al., 2020c; Chen et al., 2020]). Compared with traditional games like Go or Atari, Honor of Kings 1v1 version has larger state and action space and more complex control strategies. A detailed description of the game can be found in [Ye et al., 2020c].

Baselines. Since the action space is discrete, we compare our method with DQN [Mnih et al., 2013] and the discrete version of BCQ [Fujimoto et al., 2019] instead of BEAR. Compared with DQN, BCQ only adds one generative model to learn the distribution of D. For AQL, we add an residual network upon BCQ as the discrete version of our proposed method. We use the same network parameters and training parameters for all the baseline methods for a fair comparison.

Results. In our experiment, we evaluate the ability of the agents learned by DQN, BCQ and our method, against the internal behavior-tree-based AI in the game, as is shown in Figure 2. We have the following observations:

• Offline RL methods (BCQ and our proposed method) largely reduce the convergence time of classical off-policy method DQN. This means the generative modeling of the dataset D helps the learning of value function.
• Our proposed method further outperform BCQ. Compared with BCQ, the discrete version of our method uses the residual network in the generative modeling process, which mitigates the error of modeling the distribution of D, and boosts the performance of learned value function.

7 Conclusion

The goal of this work is to study the impact of generative modeling error in offline reinforcement learning (RL). We theoretically and empirically analyze how error propagates from generative modeling to the value function approximation in Bellman backup of off-policy RL. We propose AQL (action-conditioned Q-learning), a residual generative model to reduce policy approximation error. Our experiments suggest that AQL can help to boost the performance of offline RL methods. Our case study on complex tasks further verifies that offline RL methods can efficiently learn with faster convergence when integrated in the online process with offline learning.