Persistent Safety Set Guided Offline Safe Reinforcement Learning

1 Introduction

Safety-critical tasks such as autonomous driving [Gulino et al., 2023; Osinski et al., 2019], robotics [Gu et al., 2023], and healthcare [Fang et al., 2024] require RL algorithms [Sutton and Barto, 2020] that maximize performance while guaranteeing constraint satisfaction [Wachi et al., 2024]. A single violation can cause irrecoverable harm, making online exploration risky. In offline RL, where learning is from static datasets [Fujimoto et al., 2019], ensuring safety is even more challenging due to risks of unsafe generalization, value overestimation, and reward propagation through unsafe regions. Prior work on safe RL has mainly used constrained optimization, introducing Lagrange multipliers [Gu et al., 2024; Wachi et al., 2024; Achiam et al., 2017; Tessler et al., 2018]. However, these approaches only encourage safety in expectation. An alternative line formulates safety via reachability:

RCRL [Yu et al., 2022] analyzes worst-case constraint evolution, which BCRL [Brahmanage and Kumar, 2026] extends to cumulative constraints, while FISOR [Zheng et al., 2024] uses diffusion models to estimate safe trajectories. Inspired by control theory, these methods use control barrier functions (CBFs) to enforce forward invariance [Ames et al., 2019]. Recent work further shows that value functions can act as barrier certificates [Tan et al., 2023].

Recent surveys on learning CBFs [Wachi et al., 2024; Ganai et al., 2024] emphasize their importance for connecting verification, optimization, and learning, but also highlight challenges: (a) learning barrier certificates from limited data, (b) integrating barrier constraints into value learning, and (c) preventing safety information "leakage" through unsafe regions. Most offline safe RL algorithms [Chemingui et al., 2025b; Koirala et al., 2025] enforce safety only at policy extraction, allowing reward propagation through unsafe regions in the value critic and weakening safety guarantees. Our work addresses these issues by learning CBFs from offline data via a generalized Bellman operator and introducing principled methods to prevent safety leakage.

Related work. Recent work has connected control barrier functions (CBFs) and reinforcement learning (RL), especially in data-driven and offline settings. Neural CBFs trained from demonstrations or safe trajectories enable safety without explicit models [Harms et al., 2024; Lindemann et al., 2024; So et al., 2024]. Some approaches estimate CBFs from offline data, but often rely on conservative assumptions or are limited to specific systems [Yu et al., 2025; Tabbara and Sibai, 2025]. In offline RL, strategies like boundary-to-region supervision, diffusion regularization, and online optimization layers [Su et al., 2025; Guo et al., 2025a; Chemingui et al., 2025a] mitigate unsafe generalization, but do not provide a unified framework for learning persistent safety sets via Bellman updates. Our work addresses this gap by formulating safety as a global, trajectory-level property, propagating worst-case violations through Bellman updates, and integrating safety critics with offline value-based RL to prevent unsafe value propagation during training.

Contributions. Our main contributions are as follows:

• Generalized Bellman operator for CBF learning: We develop a framework for learning discrete time CBFs via a generalized Bellman operator, and show that sev-

eral existing CBFs are special cases. The fixed point of this operator corresponds to the largest forward-invariant safe set, providing a data-driven safety certificate.

• Forward-invariant safety enforcement: We integrate learned CBFs as an explicit safety critic in our offline safe RL algorithm, embedding them directly into actor-critic learning so that both value estimation and policy improvement are restricted to barrier-certified safe regions, preventing value propagation via unsafe regions, while still achieving high returns.
• Empirical validation: We demonstrate state-of-the-art safety constraint satisfaction with competitive returns on standard offline safe RL benchmarks.

Additional results, code and full paper with proofs are available at https://ayanroy24.github.io/GDCBF/.

2 Preliminaries

2.1 Offline Safe Reinforcement Learning

We formulate safe sequential decision making as a constrained MDP (CMDP), and subsequently address the challenges of learning from an offline dataset.

ConstrainedMDP. ACMDP is defined by the tupleM := ⟨S,A, T , r, c, γ, µ0,Clim⟩. Here, S and A denote the state and action spaces, respectively [Altman, 2021]. The transition distribution T (s′ | s, a) denotes the probability of transitioning to state s′ given state s and action a. Reward function is defined as r : S × A → R and the cost function as c : S ×A → R+. The reward discount factor is γ ∈ [0, 1), and µ0 denotes the initial state distribution. The scalar Clim represents the safety budget, defined as the maximum allowable expected cumulative cost.

The objective is to learn a policy π : S → A that maximizes the expected discounted return JR(π) subject to the safety constraint JC(π) ≤ Clim, formally stated as:

where Eπ denotes the expectation over trajectories induced by π and T . In this work, we set the safety budget Clim = 0, addressing the challenging hard constraint setting where any constraint violation is inadmissible.

Learning from Offline Data. In complex real-world domains, obtaining an accurate analytical model of the environment (e.g., transition dynamics T ) is often intractable. Consequently, we adopt the offline safe RL setting, where the agent must learn solely from a fixed dataset D = {(si, ai, ri, ci, s′i )}Ni=1 collected by one or more behavioral policies πβ, without additional environment interactions.

2.2 Discrete Control Barrier Value Functions

Expected-cost constraints in CMDPs do not prevent transient or worst-case violations. In safety-critical domains, a single violation may be unacceptable regardless of the expected cost. Control barrier functions (CBFs) provide a stronger safety mechanism by characterizing safety through forward invariance of a feasible set. Next, we adapt CBFs [Agrawal and Sreenath, 2017] to discrete-time offline safe RL.

Safety is enforced through inequalities on value functions, enabling barrier-based reasoning directly from offline datasets. We learn CBFs via a Bellman-style update rule to learn a safety-critic Qh(s, a) that serves as a safety certificate. We define the safe set C based on state-action pairs, where the function h : S × A → R serves as a safety indicator.

Definition 2.1 (Safe Set). The safe set C is defined as the set of state-action pairs satisfying the condition:

where h(s, a) represents the immediate safety indicator (e.g., distance to a safety boundary), and positive values indicate a violation.

For CMDPs, the safety indicator h(s, a) is directly derived from the cost function c(s, a). Following [Zheng et al., 2024], let h(s, a) = c(s, a) if the transition is unsafe (c(s, a) > 0), and h(s, a) = -1 otherwise. This mapping ensures that h(s, a) > 0 corresponds to constraint violations, while h(s, a) ≤ 0 indicates safety compliance.

While C captures instantaneous safety, relying solely on it is myopic, as it does not ensure safety can be maintained over time. Guaranteeing long-term safety requires identifying regions where the dynamics allow indefinite constraint satisfaction, leading to two notions: forward invariance under a given policy, and the existence of at least one safe policy, which defines the persistent safety set. For exposition ease, we assume below a deterministic MDP; stochastic setting is addressed later.

Definition 2.2. A set Sπ ⊆ S is forward invariant with respect to a policy π if, for any initial state s0 ∈ Sπ, the subsequent state trajectories generated by the deterministic transition function f : S × A → S, st+1 = f(st, π(st)), satisfy:

If Sπ is a subset of the safe states (where (s, π(s)) ∈ C), then the policy π guarantees safety for all time steps when initialized within Sπ, and thus π satisfies the CMDP safety constraint in (1).

Definition 2.3. The persistent safety set SP ⊆ S is defined as the set of all states for which there exists a policy π such that the agent remains safe indefinitely. Formally:

This set represents the maximal controlled invariant set within the safety constraints. Staying in this set enforces CMDP constraint in (1).

Explicitly computing set SP requires solving a global reachability problem [Bansal et al., 2017], which is computationally intractable [Mitchell et al., 2005]. CBFs provide a tractable alternative. By satisfying a specific rate-ofchange condition, the CBF function estimates the persistent safety set, effectively identifying the boundary of safe operation [Ames et al., 2019].

Specifically, we learn a safety-critic Qh(s, a) that serves as a certificate for persistent safety. The indicator h(s, a) captures whether an action is immediately safe, Qh(s, a) accounts for the worst-case future safety violations that may occur along subsequent trajectories. Intuitively, if Qh(s, a) ≤ 0, the state-action pair is persistently safe, meaning there exists a policy that can maintain safety indefinitely. To ensure forward invariance, we define the following condition:

Definition 2.4 (Deterministic Discrete CBF). Consider a deterministic MDP with transition function f (Definition 2.2). A function Qh(s, a) is a discrete CBF if there exists a policy π such that for all (s, a) with Qh(s, a) ≤ 0:

where α(·) is an extended class K function [Agrawal and Sreenath, 2017].

An extended class-K function α : R → R is continuous, strictly increasing, with α(0) = 0; it modulates the decay rate in Eq.5, permitting residency in the safe set for Qh ≤ 0 while enforcing a corrective decrease toward safety for Qh > 0. Hence α(q) > 0 for q > 0 and α(q) < 0 for q < 0, so α(Qh) has the same sign as Qh. A canonical choice is the linear form α(q) = λq with λ ∈ (0, 1]. Intuitively, this inequality guarantees forward invariance by restricting the evolution of Qh so that the agent is strictly prevented from crossing the safety boundary. The function α(·) modulates this restriction, effectively dictating how aggressively the policy must steer away from the unsafe region based on the current safety margin. In stochastic environments, relying on a single next state or an expectation is insufficient for strict safety guarantees. We therefore adopt a robust formulation.

Definition 2.5 (Robust Discrete CBF Condition). For a stochastic MDP defined by T (·|s, a), a function Qh(s, a) is a robust discrete CBF if there exists a policy π such that for all (s, a) with Qh(s, a) ≤ 0:

where supp(T (·|s, a)) denotes the set of possible next states (or support), and α(·) is an extended class K function.

This ensures that, even under worst-case transitions, the safety value remains decreasing (or sufficiently negative) over time. Small increases in Qh may occur due to stochasticity, but α ensures forward invariance is maintained without requiring strict decrease.

3 Learning CBFs via Bellman Updates

3.1 Generalized Bellman Operator

Learning control barrier functions from data presents several challenges in the offline reinforcement learning setting. Classical CBF formulations typically assume access to system dynamics or require evaluating all possible successor states [Ames et al., 2014], which is infeasible without an explicit model. Existing reinforcement learning approaches instantiate barrier functions using specific Bellman operators, such as hard maximum-based reachability updates [Yu et al., 2022] or additive cost-style backups [Tan et al., 2023], each inducing different trade-offs between safety and conservatism. Moreover, offline datasets provide limited transition coverage [Fujimoto et al., 2019], requiring safety propagation mechanisms that reason about worst-case future violations rather than expected behavior. These challenges motivate the need for a generalized Bellman backup framework that unifies existing formulations while ensuring convergence and forward-invariant safety properties.

To learn this function, we define a generalized Bellmanlike operator that propagates the worst-case safety constraint violation. This effectively enforces a reachability constraint: if a future state is unsafe, the current value must reflect that potential violation.

Assumption 3.1. The function Ψ : R × R → R satisfies the following properties: it is strictly increasing in its second argument. Furthermore, Ψ(h, ·) is a contraction in the second argument, i.e., |Ψ(h, v1)-Ψ(h, v2)| ≤ γ|v1-v2| with γ < 1 for all h.

Assumption 3.2. The function Ψ satisfies a consistency condition: for any h, v, if q = Ψ(h, v), then v ≤ q - α(q) for some extended class K function α. This ensures that the fixed point induces a valid decay for the safety value.

Intuitively, this condition ensures that any fixed point of the operator B automatically respects the required safety decay, guaranteeing that the learned value function acts as a valid barrier certificate.

Definition 3.1 (Generalized Reachability Bellman Operator). We define the operator B acting on the state-action value function:

where Ψ mixes the immediate safety cost with the future cost.

3.2 Theoretical Analysis

Having defined the generalized operator, we now establish its theoretical properties. Specifically, we show that under the imposed assumptions on the mixing function Ψ, the operator defines a contraction mapping in the value function space. Consequently, its iterative application converges to a unique fixed point. Crucially, we prove that this fixed point satisfies the robust discrete CBF condition, thereby providing a valid certificate for the forward-invariant safety set.

Theorem 3.1. Consider a CMDP with continuous safety indicator h(s, a). The iterative application of the operator B defined in (7) converges to a unique fixed point Q∗h . Furthermore, Q∗h satisfies the Robust Discrete CBF condition defined in (6), provided Assumptions 3.1 and 3.2 hold.

The proof of Theorem 3.1 is available in Appendix A.1.

As a consequence of the fixed-point characterization, the boundary of the persistent safety set is given by the zero levelset of the learned barrier value function. In particular, the set

defines a forward-invariant safety region, while its boundary is implicitly induced by the condition Q∗h (s, a) = 0. Thus, the safety boundary is not learned explicitly or parameterized a priori, but emerges naturally from the generalized Bellman updates.

Maximal Forward-Invariant Safe Set. As safety in sequential decision-making is inherently global and set-valued, validity requires guarantees over entire trajectories, robustness to worst-case dynamics, and compatibility with valuebased policy optimization. Moreover, in the offline setting, safety guarantees must be grounded in dataset-supported transitions rather than extrapolated dynamics. Accordingly, we establish additional properties of the fixed point Q∗h . These results show that the zero sublevel set of Q∗h is not merely forward invariant but constitutes the maximal set of stateaction pairs from which safety can be maintained indefinitely, ruling out unnecessary conservatism. We use the standard pointwise order on value functions, i.e., Q1 ⪯ Q2 if Q1(s, a) ≤ Q2(s, a) for all (s, a), and define the safe set induced by Q as CQ = {(s, a) | Q(s, a) ≤ 0}.

Lemma 3.2. A set C ⊆ S × A is forward invariant if there exists a policy π such that, for all (s, a) ∈ C and all s′ ∈ supp(T (·|s, a)), we have (s′ , π(s′ )) ∈ C. A forwardinvariant safe set C ∗ is maximal if, for any other forwardinvariant safe set ˜ C, it holds that ˜ C ⊆ C ∗ . Let Q∗h be the fixed point of the operator B defined in (7). Then the set

is the maximal forward-invariant safe set induced by B. The proof of Lemma 3.2 is available in Appendix A.2.

4 Generalized Discrete CBF Variants

The theoretical framework established in Section 3 relies on a generalized mixing function Ψ. In practice, the choice of this function determines the nature of the learned safety certificate and its optimization landscape. In this section, we discuss what are the potential instantiations of the framework.

4.1 Deterministic Case

Maximum. The simplest instantiation of our framework aligns with the Reachability Constrained RL (RCRL) approach [Yu et al., 2022]. Here, the operator propagates the maximum of the immediate safety cost and the discounted future safety value:

where, Vh(s′ ) = mina′ Qh(s′ , a′ ). This corresponds to Ψ(h, v) = max(h, γv). The discount factor γ serves to decay the safety value over the horizon; specifically, it causes the safety margin of safe states (negative values) to degrade towards zero over time, while diminishing the magnitude of distant unsafe violations (positive values). While this operator strictly enforces the safety condition, it can suffer from value explosion if the horizon is long and costs are additive.

Smooth. The FISOR [Zheng et al., 2024] algorithm employs a convex combination to smooth the transition between the immediate cost and the future value:

This corresponds to Ψ(h, v) = (1 - γ)h + γ max(h, v). By incorporating the term (1-γ)h(s, a), this operator effectively anchors the value estimate to the immediate safety cost. This anchoring ensures that the safety critic remains responsive to local constraint violations and stabilizes the learning process by balancing the immediate feedback with the bootstrapped lookahead.

Additive. The "Your Value Function is a Barrier Function" framework [Tan et al., 2023] proposes an additive formulation where the Bellman operator aggregates the immediate safety cost and the discounted future value:

This corresponds to the linear aggregator Ψ(h, v) = h + γv. Unlike the maximum-based operators, this formulation resembles the standard Bellman equation for cumulative cost. By interpreting the value function itself (shifted by a constant) as the barrier function, this approach ensures that the safety certificate is learned via standard value iteration techniques, provided the immediate cost h(s, a) is designed to reflect the distance to the safety boundary.

Proposition 4.1. The Maximum, Smooth, and Additive update rules all satisfy Assumption 3.1 (Monotonicity and Contraction) with contraction factor γ. Furthermore, they satisfy the robust CBF condition (Assumption 3.2) with appropriate choice of α.

The proof of Proposition 4.1 is available in Appendix A.5.

4.2 Stochastic Case

The deterministic operators discussed above assume a single possible outcome for each action, which is often violated in real-world systems with aleatoric uncertainty or unmodeled dynamics. In such non-deterministic environments, the agent must satisfy the Robust Discrete CBF condition (Definition 2.5), which requires bounding the maximum violation over all possible next states: maxs′ Vh(s′ ).

In the offline setting, we lack an explicit transition model to query the set of reachable states. To overcome this, we propose to employ an asymmetric aggregation [Hansen-Estruch et al., 2023] to approximate this worst-case future violation directly from the data distribution. We denote this asymmetric aggregation as Eτ , where the parameter τ can be adjusted to control the level of conservativeness when approximating the maximum. Common choices for such objectives include expectile regression, percentile (pinball) loss, or Gumbel regression [Garg et al., 2023]. By tuning the parameter τ, we can estimate the upper tail of the future safety values observed in the dataset.

In practice, we apply the asymmetric aggregation outside the aggregator function Ψ to handle stochasticity in the transitions (s, a, s′ ). The update rule becomes:

As τ → 1, the expectile Eτ approaches the maximum operator, it also approaches the worst-case max over the transition support as seen in Fig.2, thereby recovering the Robust Discrete CBF condition (Definition 2.5) which requires bounding the worst-case future violation. Note that Eq.11 adapts Eq.7 to the stochastic case: a direct max over transitions is impractical in offline data (identical (s, a) pairs rarely repeat), so asymmetric regression estimates the upper tail as a data-efficient surrogate for worst-case estimation under limited coverage. Inmany practical implementations, Q-learning employs a Huber or L1 loss [Mnih et al., 2013] to handle outliers and stabilize training. The corresponding asymmetric variant for these robust losses is the percentile loss. Therefore, we utilize the percentile loss to implicitly approximate the robust target maxs′ Vh(s′ ) over the transition distribution, thereby enforcing safety against stochasticity in dynamics.

5 Utilizing Learned Persistent Safety Set for the Value Function Estimation

As illustrated in Figure 1, enforcing safety at the value function level prevents reward propagation through unsafe intermediate states-a common issue in prior works such as FISOR [Zheng et al., 2024] that apply safety constraints only during policy extraction. We adopt an Implicit Q-Learning (IQL) [Kostrikov et al., 2022] framework for offline RL to learn the optimal value function from the static dataset D, while enforcing the persistent safety.

Safety-Aware Joint Bellman Target. A core challenge in offline safe RL is ensuring value estimates reflect safety constraints, even when the dataset contains unsafe trajectories. Discarding unsafe transitions is undesirable because they often contain useful information, including safe segments. To address this, we first learn the safety critic Qh using the chosen CBF instantiation and then use it to define a modified Bellman target. We propose a joint Bellman target for the reward Q-function, QR, which explicitly handles safe and unsafe transitions based on learned persistent safety. The target yR(s, a, s′ ) is defined based on the safety status predicted by our learned CBF Q∗h :

where rmin = min(s,a,r,c,s′ )∈D r is the minimum reward observed across the entire offline dataset (a single datasetwide constant, not a per-(s, a) minimum). For safe actions (Q∗h (s, a) ≤ 0), we update using the standard Bellman backup with the value function VR. For unsafe actions

Algorithm 1 Barrier-Guided Offline Actor-Critic (CBF)

Require: Offline dataset D = {(s, a, r, c, s′ )}, discount γ, expectiles (τr, τh), temperature β, target rate η, samples N

• 1: Initialize actor πθ, reward criticQωr , value V ψ r , safety criticQϕh , safety value V ν h , and target networks
• 2: for each gradient step do
• 3: Sample a minibatchM= {(s, a, r, c, s′ )} ∼ D
• 4: Barrier target: yh ←Ψ(c, Vh(s′ )) for (s, a, s′ ) ∈M
• 5: Safety value update:

ν ← arg minν EM[Lτh

(Qϕh (s, a) - V ν

• 6: Safety critic update: ϕ ← arg minϕ EM[∣∣∣Qϕh (s, a) - yh ∣∣∣]
• 7: Safety-aware reward target (Eq.(12):

• 8: Reward critic update:

ω ← arg minω EM[(Qωr (s, a) - yr)2 ]

• 9: Reward value update:

• 10: Policy update: w ← exp(β(Qωr (s, a) -V ψ r (s))); θ ← arg minθ EM[-w log πθ(a|s)]
• 11: Update target networks
• 12: end for
• 13: Evaluation: sample {ai}Ni=1 ∼ πθ(·|s),
• 14: return a⋆ = arg minai Qh(s, ai)

(Q∗h (s, a) > 0), we assign a penalty term. This penalty consists of the worst-case discounted return rmin 1-γ minus the magnitude of the safety violation Q∗h (s, a). This ensures that unsafe actions have strictly lower Q-values than safe actions, effectively creating a "soft" barrier in the value landscape.

We define reward leakage as the propagation of reward through unsafe transitions during Bellman updates, causing the reward critic to overestimate values for policies that violate safety. Our safety-aware target (Eq. 12) directly penalizes such transitions during training, preventing leakage. In contrast, FISOR [Zheng et al., 2024] enforces safety only at policy extraction, allowing value propagation through unsafe regions. Enforcing safety at the value level ensures the Bellman operator converges to a safety-consistent fixed point as shown in figure 3 and the toy example in Appendix D. The reward Q-function is updated by minimizing the Mean Squared Error (MSE) loss:

Value FunctionEstimation via Expectile Regression. We then learn the state-value function VR(s) via expectile regression over the learned QR values:

By utilizing the safety-aware QR, the expectile regression (with τ > 0.5) naturally approximates the value of the best safe actions in the dataset distribution. In states where both

h (s))]

safe and unsafe actions are present, the high penalty on unsafe actions ensures they fall below the expectile, causing VR(s) to converge towards the value of the safe actions. In states where only unsafe actions are available, VR(s) correctly reflects the low, penalized value, signaling the agent to avoid such states. In practice, we can either learn the CBF as a pre-training step and then learn the reward critic QR, or learn both in parallel. The latter approach aligns more closely with standard offline RL, which typically extracts the policy while learning the Q-value. Following this approach, we learn all three components-the CBF Qh, the reward critic QR, and the policy π-concurrently, as presented in the algorithm pseudocode.

Pseudocode and Computational cost. Algorithm 1 summarizes our method where action-value functions (Qh,Qr) are parameterized to support Bellman updates and policy improvement, while value functions (Vh, Vr) are also learned via function approximation but optimized using expectile regression to provide stable targets. At each step, we fit the safety critic Qh using a Bellman-like update, then use it to filter transitions for the reward critic Qr: safe transitions use standard bootstrapped targets, while unsafe ones receive a pessimistic penalty based on rmin. The policy is improved via advantageweighted regression. This approach prevents reward propagation through unsafe states and ensures efficient, stable offline training. On a single RTX 2080 Ti GPU, 500k gradient steps require about 20 minutes.

6 Experiments

Goal of our experimental evaluation is:

1. Can a learned discrete control barrier function (CBF) effectively enforce hard safety constraints in the offline setting while obtaining high rewards ?
2. Assess how different deterministic CBFs-CBFsmooth Eq.(9), CBFadditive Eq.(10), and CBFmaximum Eq.(8)-perform in the offline setting.
3. Does the robust smooth barrier update (section 4.2), utilizing an asymmetric loss, provides a safety advantage over the standard version in the presence of uncertainty?
4. Ablation studies to analyze the algorithm's sensitivity to hyperparameters (see Appendix B.2).

Evaluation Setup. We evaluate on diverse safetycritical continuous control tasks (Safety Gym, Bullet Gym, MetaDrive) using the same experimental setup as FISOR [Zheng et al., 2024]. Rewards are normalized by the dataset's min and max, and costs by a predefined limit. A policy is considered safe if its normalized cost does not exceed one. Further details are in Appendix C.

Comparison with Baselines. We compare our method to safe offline RL baselines: CAPS [Chemingui et al., 2025b], LSPC [Koirala et al., 2025] (soft constraints), and CCAC [Guo et al., 2025b], FISOR [Zheng et al., 2024] (hard constraints). As shown in Table 1, methods enforcing safety only at policy extraction (e.g., CAPS, CCAC, LSPC) often achieve high rewards but violate constraints. In contrast, our approach (Algorithm 1) maintains near-zero cost across tasks, enforcing hard safety while achieving top rewards among safe policies. This is especially clear in recovery-heavy tasks, where restricting value backups to the barrier-certified set preserves high returns and prevents unsafe transitions. Thus, integrating a learned discrete CBF into value critic learning (Section 5) enables strict safety without sacrificing performance. FISOR and our method both satisfy the cost criterion, but our safety-aware Bellman target yields more accurate value estimates, supporting both high reward, safety.

Comparison of CBFs. A comparison of different barrier formulations is available in Appendix B.1, which reveals distinct trade-offs. For instance, CBFmaximum tends to optimize for reward, while CBFsmooth yields the most stable results. This distinction highlights that the propagation of safety information is a decisive factor in offline safe RL.

Robustness via Asymmetric Losses. Figure 2 compares the standard smooth (deterministic) barrier update (Eq. (9)) with its robust variant (Eq. (11)). To test robustness, we modify the DSRL dataset by mislabeling 70% of high-cost trajectories as safe using the DSRL's built-in functionality. We evaluate both deterministic and robust versions of Smooth CBF across velocity control and MetaDrive tasks. The robust version utilizes an asymmetric loss (percentile) with τ = 0.6, while the standard version uses the standard L1 loss. RobustSmooth consistently reduces normalized cost compared to the Deterministic-Smooth version. Particularly in velocity tasks, Robust-Smooth achieves nearly zero cost whereas the Deterministic-Smooth version incurs high costs. This improvement is achieved with only a modest reduction in normalized reward, demonstrating that asymmetric losses effectively estimate a persistent safety set under uncertainty. Importantly, these trends are consistent across all tasks, suggesting that improvements arise from stochasticity-aware safety estimation rather than task-specific tuning. Although, the robust variant uses asymmetric losses to approximate worstcase transitions, reducing costs on average in stochastic environments, but for hardense/hardsparse, limited data coverage causes Eτ to underestimate extreme violations, leading to less conservative behavior and slightly higher costs than the deterministic formulation. Complete results across all velocity control, MetaDrive environments are in Appendix B.3.

Reward Leakage Analysis. To empirically evaluate reward leakage, we save the fully trained model and randomly sample 10K state-action pairs from the offline dataset. For each pair, we compute the safety critic Qh(s, a) and reward critic Qr(s, a) using the saved model, and then plot Qr(s, a) versus Qh(s, a) to verify that, for Qh(s, a) > 0, the reward critic remains low, indicating that reward does not propagate through unsafe transitions. Figure 3 confirms that CBFsmooth approach effectively blocks reward propagation through unsafe regions, ensuring that high rewards are only assigned to safe state-action pairs and providing strong safety guarantees during value learning.

7 Conclusions

We propose an offline safe RL framework that learns discrete CBFs via a generalized Bellman backup, unifying prior CBF methods with convergence guarantees. We further introduce a robust asymmetric variant for worst-case transitions and a safety-aware Bellman target to prevent reward leakage, together with a value-based offline RL method for safe policy extraction. Across continuous control and autonomous driving benchmarks, our method satisfies constraints while achieving competitive or superior reward.

Ethical Statement

There are no ethical issues.

Contribution Statement

Ayan Choudhury and Janaka Brahmanage contributed equally to this work.