ROAD: Adaptive Data Mixing for Offline-to-Online Reinforcement Learning via Bi-Level Optimization

1 Introduction

Deep reinforcement learning (RL) has achieved remarkable access in various domains [Wang et al., 2024; Mazyavkina et al., 2021], yet its deployment in real-world applications is fundamentally restricted by the prohibitive sample complexity of online learning methods [Ladosz et al., 2022; Kumar et al., 2023], especially in safety-critical scenarios

∗An extended version of this paper with all appendices is available at https://arxiv.org/abs/2605.14497.

(e.g., healthcare) or costly environments (e.g., autonomous driving) [Yu et al., 2021; Zhao et al., 2025]. Offline RL offers a promising alternative by leveraging previously collected datasets to extract a policy without environment interaction [Levine et al., 2020]. However, strictly offline learning is inherently limited by the quality and coverage of the static datasets, failing to reach a performance comparable to agents trained online. To bridge this gap, offline-to-online (O2O) RL has emerged as a new paradigm, aiming to initialize an agent with a safe offline policy (the offline phase) and subsequently refine it via direct online interaction (the online phase) [Nair et al., 2020]. This paradigm seeks to maintain the safety of offline policy while enabling the plasticity to explore and improve beyond the offline dataset.

Under such a twofold objective, effective O2O training is non-trivial. The transition from offline pretraining to online fine-tuning introduces severe distribution shift [Lee et al., 2022; Huang et al., 2025]. Naively concatenating offline and online RL algorithms typically results in a catastrophic "unlearning" phenomenon, characterized by a significant performance dip at the beginning of online fine-tuning [Nakamoto et al., 2023]. A majority of works balance pessimism from offline RL [Kumar et al., 2020; Kostrikov et al., 2022] and optimism required for online RL [Haarnoja et al., 2018] through Q-value calibration [Nair , 2020; Nakamoto et al., 2023].

However, there have been limited works on addressing the distribution shift directly through data-level distribution alignment. Existing data-centric solutions largely rely on static strategies [Nakamoto et al., 2023; Zhang et al., 2023] or fixed heuristics [Lee et al., 2022; Liu et al., 2025b] to mix offline and online dataset. However, we argue that there is no universally optimal data mixing strategy. Instead, the optimal data replay pattern is tailored to the quality of the offline data, the environment properties and even the stage of training. For instance, a high proportion of offline data might be necessary in early training to stabilize updates, whereas later stages might benefit from prioritizing online experience to correct the bias from offline data.

In this paper, we investigate the mechanism by which data mixing strategies influence online fine-tuning, characterizing data mixing as a reshaping of the Q-learning loss landscape. From this perspective, we identify an objective misalignment problem in offline-to-online settings, where the conventional Q-learning objective of minimizing the Bellman error does not suffice to guarantee RL performance.

To address this objective misalignment issue, we propose reinforcement learning with optimized adaptive data-mixing (ROAD), a dynamic, plug-and-play framework that adaptively determines the optimal data mixing strategy. We fundamentally reframe the data selection problem as a bi-level optimization process in the functional space of data mixing strategies, where the inner-level updates follow standard Qlearning, and the outer-level takes data mixing as a metadecision problem to maximize the expected performance of the online policy, thereby aligning Q-learning with the ultimate goal of maximizing RL performance. A solution is provided by computing the gradient of the outer-level objective.

To further deal with the overestimation bias of the outerlevel objective and thus bridge the gap between theory and practice, ROAD constructs a surrogate as an unbiased estimator for the outer-level objective. A practical algorithm utilizing a sliding-window multi-armed bandit (MAB) to optimize the data mixing ratio is proposed to optimize this surrogate objective. Intuitively, ROAD selects an arm that maximizes the performance increment in trusted regions where the uncertainty of the Q-function estimator is negligible and mitigates training instability by suppressing value overestimation. In summary, our contributions are as follows:

• We identify an objective misalignment problem in datacentric O2O RL and solve this misalignment by formulating it as a bi-level optimization, where the datamixing strategy is considered a meta-decision variable.
• We propose ROAD, a plug-and-play framework that approximates the bi-level optimization theory with a computable, unbiased surrogate objective and a MAB mechanism optimizing data mixing ratios in real-time.
• We evaluate ROAD across extensive benchmarks and the results show that ROAD outperforms static and heuristic-based data selection strategies, ensuring training stability and asymptotic performance.

2 RelatedWorks

The primary challenge of O2O RL lies in the mismatch between the principles underlying the two phases. Specifically, offline algorithms feature restrictive policy constraints and pessimistic Q-value estimation [Kumar et al., 2020; Kostrikov et al., 2022] to ensure conservatism, which inherently limits the exploration capabilities during online finetuning [Xie et al., 2021; Song et al., 2022; Wagenmaker and Pacchiano, 2023], resulting in slow performance gains. To mitigate this, transitional frameworks have been proposed. Some algorithms have adopted implicit or dynamic constraints [Nair et al., 2020; Zhao et al., 2022; Lee et al., 2022] which naturally loosen in the online phase. Similarly, CalQL [Nakamoto et al., 2023] conducts offline Q-value estimation in a calibrated rather than pessimistic manner; WSRL [Zhou et al., 2025] introduces a separate warm-start phase to calibrate Q-value and then discards the offline dataset. Another line of work embraces the discrepancy between offline and online settings [Liu et al., 2025a]. APL [Zheng et al.,

2023] employs pessimistic updates on offline data and optimistic updates on online data; PEX [Zhang et al., 2023] maintains separate policies for offline and online learning; MOORL [Chaudhary et al., 2025] treats the O2O RL problem as a meta-learning problem with two distinct tasks, offline and online learning.

2.1 Offline Data Replay for Offline-to-Online RL

While algorithmic consistency is crucial, the strategy to address data-level distribution shift plays a decisive role in ensuring stability and plasticity in the online fine-tuning phase. Although some argue that offline data can be completely discarded in online learning [Zhou et al., 2025], its adaptiveness to various environments is questioned [Li et al., 2025a]. The predominant solution to offline data utilization during online interaction is to replay offline data in the online replay buffer.

Static Mixing Strategies Early attempts initialize the replay buffer with offline dataset [Hester et al., 2018; Vecerik et al., 2017], which proves unstable [Hansen et al., 2023]. Instead, the current mainstream is to sample independently from two data sources with a mixing ratio. QT-Opt gradually increase the online data ratio from 1% to 50% [Kalashnikov et al., 2018]. Cal-QL, RLPD and PROTO implement symmetric sampling (a fixed 50%-50% split between offline and online data) [Nakamoto et al., 2023; Ball et al., 2023; Li et al., 2023], which is also adopted in concurrent researches [Wu et al., 2025; Li et al., 2025b]. While simple, these static strategies are inherently suboptimal for lack of context awareness and environment adaptiveness.

Heuristic-based Data Selection Recognizing the limitations of fixed mixing ratios, there have been works toward selective sampling. Balanced replay and following works prioritizes "near on-policy" offline samples [Lee et al., 2022; Liu et al., 2025b; Song et al., 2025]. Despite mitigating distribution shift, near on-policy selection heuristics reduces data selection to a form of online data augmentation, limiting the algorithm's ability to fully leverage the environmental information encoded in the offline dataset. Our empirical findings, together with prior analytical works [Li et al., 2025a], indicate that no single sampling strategy generalizes universally, which motivates an adaptive, context-sensitive strategy that adapts to learning dynamics and environmental feedback [Nakamoto et al., 2023; Zheng et al., 2023].

3 Bi-Level Optimization for O2ORL

In this section, we formulate data mixing as an optimization problem. From a functional perspective, we offer a novel perspective on how different data mixing strategies influence the training dynamics. Under this perspective, we argue that an optimal mixing strategy should be derived by aligning the Qlearning objective with the true RL objective. Based on this formulation, we present an idealized optimization algorithm as a theoretical solution.

3.1 Objective Misalignment in O2O Fine-Tuning

We formulate the O2O RL problem within the Markov Decision Process (MDP) framework M = (S,A, p, r, γ). The online fine-tuning begins with a static offline dataset Doffline generated by a behavior policy πβ and an offline initialization for the policy πoff and the Q-function f off in a function class F. Let the initial state distribution be µ ∈ ∆(S) and ρ be any trajectory, then the goal is to maximize the cumulative value function, denoted by J(π) = Es∼µ [V π (s)] = Eρ∼π,s∼µ [∑∞t=0 γ t rt|s0 = s].

Online fine-tuning via fitted Q-iteration. Standard valuebased online fine-tuning algorithms typically follow a fitted Q-iteration (FQI) scheme [Nakamoto et al., 2023]. At each iteration k, the agent collects an online data batch Dk with the online policy πk . Then the Q-value iteration is performed on a mixed distribution dsample built with both the offline dataset and online replay buffer. Specifically, the Q-function updates with Equation (1) where T is the Bellman operator defined as T f(s, a) = r(s, a) +γEs′ [maxa′ f(s′ , a′ )].

Crucially, the construction of the mixed distribution is formulated via a data mixing strategy g ∈ G, defined as a mapping from online and offline data sources to a sampling distribution, where G is the function space of all such strategies.

The loss landscape perspective on data mixing. Equation (1) reveals a critical insight that Q-learning is essentially projecting the Bellman target onto the representable function space F, i.e., f k+1 ← ΠF,dsample g (T f k ). Here ΠF,dsample g denotes the projection operator defined by weighted L2-norm ∥·∥dsample g . This formulation makes explicit the mechanism through which data mixing strategies affect online RL finetuning, as depicted below. The data mixing strategy governs the geometry of the projection. By altering the sampling density over the state-action space, different mixing strategies define distinct loss landscapes, which further steers the optimization toward different optima f ∗ (g). The induced variation in Q-function updates further yields different online policies and, ultimately, different RL performance levels J(π).

The misalignment between objectives. The dependency chain of variables above demonstrates that the primary obstacle to identifying ideal data mixing stems from its indirect influence on the training dynamics, which gives rise to an objective misalignment problem. Specifically, contrary to standard online RL, minimizing the Bellman error in online fine-tuning does not guarantee maximizing the policy return due to the extra variable g.

Existing approaches dependent on static mixing ratios and heuristic selection effectively lock the projection geometry based on fixed priors, assuming that a particular landscape is universally beneficial. However, this assumption is fundamentally flawed because the relationship between the projection geometry and online performance is dynamic, dependent on the available data, the online policy and the stage of training. As an example, geometry required for early-stage stability (anchoring to offline data to prevent catastrophic forgetting) is the inverse of that required for asymptotic plasticity (emphasizing online samples to correct offline bias). The fixed priors cut off the explicit feedback from the online policy performance, failing to satisfy these mutually exclusive requirements simultaneously.

To resolve this misalignment, we transcend static rules and treat g as a meta-decision variable to study its effect on the policy performance (our final objective) rigorously. Since aligning the Bellman objective with RL performance necessitates precise control over the loss landscape across the stateaction space, we abstain from parameterizing g and formulate it directly as a functional meta-decision variable.

3.2 The Bi-Level Optimization Formulation and The Theoretical Solution

To address the objective misalignment problem, we reframe data mixing as a dynamic meta-decision process, solving an optimization problem in the space of mixing strategies to dynamically align the Q-learning objective with policy performance maximization. Notably, this optimization is inherently hierarchical because the final objective (policy performance) depends on the learned Q-function, which is itself an optimum for minimizing the Bellman error. The constraint from an optimal inner Q-learning loop naturally casts the data mixing problem into the framework of bi-level optimization.

Formally, the inner loop is defined as standard Q-learning updates, which we treat as a fixed subroutine. The objective J k in (g, f) is exactly the Bellman error to be minimized on the mixed dataset, as is defined in Equation (1). In order to dynamically determine the optimal mixing, the outer-loop is expected to make a decision on the meta-variable g within each FQI iteration, maximizing the expected return J(π) of the induced policy. Using the chain of variable dependency, we can express the objective as a function of g, leading to the objective in Equation (3).

Here we assume a differentiable policy mapping from the inner-level optimal f ∗ to the behavior policy π(f ∗ ) to ensure that the gradient exists.

The theoretical solution. Inspired by recent breakthroughs in bi-level optimization theory [Petrulionyt˙e et al., 2024], we derive a theoretical solution to the proposed setting. Within each FQI iteration, we solve the optimization via gradient ascent. The critical part is the gradient calculation of the outerlevel objective demonstrated in Equation (4).

Here wg denotes the density ratio between the distribution of a policy induced by the online policy and its advantage function, and the sampling distribution; the mixed Fr´echet derivative ∂g,fJin evaluates the sensitivity of Bellman error to the data mixing strategy. Intuitively, Equation (4) implies that the ideal g∗ induces a weighted on-policy distribution biased towards the loss-sensitive samples.

4 ROAD: Towards Practical RL Fine-Tuning

Despite an idealized procedure for optimal data mixing strategy in Section 3.2, directly solving for an optimum with Equation (4) is intractable in online settings. Consequently, in this section, we present a practical algorithm for the bi-level optimization, ROAD, for online RL fine-tuning.

4.1 Bridging Theory with Practice

The primary obstacle to implementing theoretical gradient ascent problem are the estimation bias of the Q-function and severe numerical instabilities in gradient computation. First, in practice, J(π(f ∗ )) is a biased estimation for the outer-level objective because π is optimized against a Qfunction f ∗ characterized by uncertainty. Maximizing this biased objective inevitably misguides the optimization process, which necessitates an unbiased estimator of J(π). Furthermore, computing this gradient involves multiple density estimations including that for a rapidly evolving online policy dπ . These estimations are inherently unstable on online data, and the arithmetic operations further aggravate this instability. Meanwhile, the high variance associated with the mixed Fr´echet derivative term easily mislead the gradient direction in sampling-based estimation. Consequently, to make unbiased estimation on the outer-level objective and meanwhile ensure computational feasibility, we adopt a zero-order optimization approach, employing a MAB mechanism to search for an optimal data mixing strategy. We establish our practical algorithm on the following two key assumptions.

Assumption 1. The offline dataset offers sufficient number of samples within its support, leading to negligible uncertainty. Assumption 2. The training dynamics varies slowly.

Assumption 1 posits that offline data serves as an anchor. Within this "trust region", it is assumed that uncertainty does not significantly distort the perceived relative value of actions. Meanwhile, Assumption 2 is the fundamental premise for modeling non-stationary environments using MAB. It allows for a sliding-window approach to evaluate the performance of data mixing strategies.

An unbiased objective estimator. An unbiased estimation for the outer-level objective is shown in Equation (5). The empirical estimation error ϵ = ˆ f -f is assumed to be a zeromean random field that is independent of f and is characterized by a continuous covariance kernel K, which effectively measures the epistemic uncertainty.

where ∥R3∥ = o (∥ϵ(s)∥3 ) is the third-order remainder and the dependency on s is omitted for brevity.

Linearized mixing strategies. To tackle computational intractability, we restrict the search space to linear combinations of offline and online distributions, parameterized by an offline data replay ratio m ∈ [0, 1]. Then the mixed data distribution is dsample m (s, a) =mdoff (s, a)+(1-m)don (s, a). To ensure compatibility with the MAB search mechanism, the candidate set of mixing ratios Λ is a fixed discrete one.

4.2 The Practical Algorithm

In this section we construct a robust surrogate objective utilized as the reward within a sliding-window MAB framework [Liu et al., 2024]. Our approach relies exclusively on readily available statistics and avoids complex auxiliary techniques such as density estimators or Q-ensembles [Lee et al., 2022], incurring virtually no additional computational cost.

The surrogate objective. By approximating the unbiased outer-level objective with statistics directly from the training process, we formulate our surrogate objective as Equation (6), where κ is a hyperparameter. In this surrogate, ∆off is the empirical estimator of RL performance improvement with respect to the offline behavior policy. By employing this advantage-like mechanism, we reduce the variance of performance estimation, leading to a stable MAB reward. Meanwhile, ∆on is an approximation of the overestimation bias at early stages, stabilizing the training process.

Rq = ∆off - κ∆on where

Theorem 1. When the Q-function f is smoother than the estimation error ϵ, and the policy improvement follows a policy gradient step with small learning rate, Rq approximates of the unbiased objective with small variance.

Proof sketch. The performance improvement term is approximated by ∆off using the performance difference lemma [Kakade and Langford, 2002], while the overestimation bias equals ∆on (signal) minus a true advantage term (noise). We formalize the smoothness of ϵ and f via their characteristic lengths, revealing that the signal-to-noise ratio of this overestimation bias estimate scales linearly with the ratio of these lengths and decays during training. Therefore, ∆on yields a precise bias estimate initially. As training progresses, the magnitude of overestimation bias itself diminishes significantly so that ∆on is effectively dominated by ∆off, introducing negligible adverse effects to the training process.

Intuitively, maximizing ∆off encourages policy improvement in the offline "trust region", while minimizing ∆on stabilizes online learning and prevents unlearning. Though seemingly counter-intuitive, ∆on functions analogously to a min-max regularization by constraining aggressive policy updates and preventing the policy from exploiting the Qfunction bias.

Implementation of ROAD. With a practical surrogate objective, we describe our algorithm ROAD as follows. ROAD works under a certain period (e.g., an episode or a certain number of steps) and maintains a recording list for the arm pulled and the surrogate objective (mk ,Rk q ) for each period k. When a period begins, ROAD selects a mixing ratio by maximizing the upper confidence bound (UCB) as is shown in Equation (7) and fixes the ratio during the period. In order to model the non-stationarity in the training dynamics, ROAD adopts a sliding window of size τ , which means that we select the best arm using only results from the recent τ periods.

Here ¯Rq,m is the average of recorded Rq values for mixing ratio m within the sliding window; Nk(τ, m) denotes the total number of times m has been selected up to period k.

5 Experiment

We study how well ROAD can facilitate efficient training in the online phase with adaptive data mixing. To this end, we compare ROAD with several other data replay methods on a variety of offline-to-online RL benchmark tasks with various offline datasets. We also study the behavior of ROAD in adaptively adjusting its data replay pattern in various environments. Finally, we perform empirical studies to investigate the robustness of ROAD under different parameter settings.

O2O RL tasks and settings. Our benchmarking tasks and datasets include: (1) AntMaze tasks from D4RL [Fu et al., 2020] which involves navigating an ant quadruped robot to a predetermined goal location within a maze; (2) Gym MuJoCo Locomotion tasks [Todorov et al., 2012; Kostrikov et al., 2022] that leverage the MuJoCo physics simulation engine targeted for robotic control with distinct morphologies and objectives; (3) FrankaKitchen tasks from D4RL [Nakamoto et al., 2023] where a 9-DoF Franka robot is controlled to reach a target configuration within a kitchen setup.

Implementation details for ROAD. We examine the effect of ROAD and the baseline data replay approaches with multiple popular offline-to-online RL algorithms, including IQL [Kostrikov et al., 2022], PEX [Zhang et al., 2023], CQL [Kumar et al., 2020], and Cal-QL [Nakamoto et al., 2023], to illustrate the robustness of ROAD when integrated with various offline phase implementations. Due to the space limits, we report only the IQL-based results (Section 5.1). The candidate set Λ is {0.1, 0.2, 0.3, 0.4, 0.5}, and the hyperparameter κ = 1. The exploration parameter c = 2.0 in all cases to encourage adaptation to the varying training dynamics. The sliding-window size is τ = 1, 000 for all tasks except τ = k for Antmaze because these tasks feature sparse rewards which leads to high variance in estimating Rq.

Baselines We compare ROAD with the following baseline data mixing strategies: (i) Fixed [Zhang et al., 2023; Nakamoto et al., 2023] mixing ratios conduct online finetuning on a linear combination of online and offline distributions. We run the experiments across all fixed ratios in Λ to compare ROAD against the best one. (ii) Decreasing ratio adopts the heuristic that offline data stabilizes training at early stages, and thus decreases the mixing ratio linearly from 0.5 linearly to 0.1 (we do not take 0 as minimum for better performance). (iii) Balanced Replay (BR) [Lee et al., 2022] adopts another family of heuristics that "nearly onpolicy" samples help stabilize training. (iv) Uniform ratios select mixing ratios uniformly within the candidate set.

5.1 Empirical Results

ROAD shows solid performance and adaptivity across various environments. Table 1 provides an analysis of ROAD's performance using normalized metrics with IQLbased offline phase as an example. Fixed mixing ratios have a performance heavily dependent on the chosen ratio. While some environments are less sensitive, others exhibit pronounced differences. Heuristic-based strategies like Decreasing and BR offer improved robustness to data quality variations but lack consistent superiority. In contrast, ROAD presents consistently strong or comparable performance across all tested environments, emerging as the most stable and adaptive method evaluated. The comparison against Uniform arm selection demonstrates the efficacy of our proposed surrogate Rq.

ROAD shows adaptivity across various offline data qualities and RL algorithms. In our benchmarking environments, ROAD showcases its ability to handle diverse offline data qualities. The performances of baselines exhibit varying performance depending on dataset quality while struggling to adapt to changing data quality. For example, when an initially inferior offline policy undergoes training and outperforms the offline dataset, it is generally intuitive to switch the data mixing strategy. By contrast, ROAD consistently matches or exceeds baseline performance across different dataset qualities. This adaptability extends to various RL algorithms, including PEX, Cal-QL and CQL. ROAD also works effectively with RLPD [Ball et al., 2023], where no pretraining is involved. These results highlight ROAD's flexibility and robustness in diverse settings, adapting seamlessly to varying algorithms and offline data conditions.

5.2 Understanding ROAD

The dynamics of Rq. Our surrogate objective Rq leverages in-training statistics to capture the training dynamics and guide data mixing decisions accordingly. To intuitively illustrate the physical interpretation of our bi-level optimization theory, we construct a simple MDP and a corresponding offline dataset, and plot the curves of ∆off and ∆on throughout the online fine-tuning phase. As is depicted by Figure 3, ∆off consistently acts as a proxy for policy improvement. Conversely, early fine-tuning is safely constrained by minimizing ∆on. ∆on later decays in scale, leaving it as a noise term in Rq and thus preventing it from suppressing true policy improvement. This allows ROAD to ensure training stability while simultaneously maintaining asymptotic performance.

The behavior of ROAD. To analyze how ROAD adapts mixing ratios of offline and online data across environments, we visualized its decisions during training using heatmaps. These heatmaps, shown in Figure 2, capture the evolution of mixing ratios for different environments and offline datasets. Specifically, different environments witness different type of data mixing strategies. In some cases, ROAD converges to a fixed ratio, while in other cases, ROAD adapts its strategy to the altered training dynamics. By exploring diverse ratios early in training, ROAD enables policies to become more robust to varying state-action distributions, leading to improved learning outcomes and more effective integration of data sources.

5.3 Ablation and Parameter Sensitivity Study

The hyperparameter κ. This hyperparameter effectively balances aggressive policy improvement and conservative offline data utilization. Two extremes would be κ = 0 and κ → ∞ corresponding to ablating the contributions of the offline and online components of relative policy quality. Results in Table 2 demonstrate that the full ROAD framework, combining both offline and online components, consistently outperforms settings that rely solely on one component. However, ROAD demonstrates low sensitivity to κ provided that κ falls within a moderate range. In these cases, ROAD works as expected, functioning as a stabilizer initially (dominated by ∆on) and optimizes asymptotic performance later on (dominated by ∆off).

The sliding-window size τ and exploration parameter c. Two critical parameters in balancing the exploration and exploitation of the ROAD framework are the exploration parameter c and the sliding-window size τ . To explore the effects of these parameters, we conducted experiments on the Halfcheetah environment with various combinations of c and τ , as detailed in Table 3. The results demonstrate that ROAD's performance remains robust across different parameter settings, highlighting its resilience to parameter variations. Typically, setting τ = 1000 better captures the slowly-varying training dynamics in most environments, leading to better asymptotic performance. This indicates ROAD's broad adaptability to different environments and offline datasets.

6 Conclusions, Limitations and FutureWork

In this work, we provide a novel perspective on how data mixing affects offline-to-online RL fine-tuning. Under this perspective, we identify the objective misalignment problem, where minimizing the Bellman error does not translate to maximized policy performance. To address this misalignment, we propose Reinforcement Learning with Optimized Adaptive Data-mixing, a novel, plug-and-play framework that formulates data mixing as a bi-level optimization problem. By treating data mixing strategy as a meta-decision variable, ROAD dynamically aligns the inner-level Q-learning objective with the goal of optimizing policy performance.

Our theoretical analysis of the optimization problem yields a gradient-based solution. Central to bridging the theory and practice is the derivation of a theoretically grounded surrogate objective Rq which approximates the unbiased estimator of the outer-level objective. This surrogate encourages policy improvement in trusted regions (via ∆off) and suppresses value overestimation (via ∆on). Extensive empirical evaluations demonstrate that ROAD consistently outperforms existing methods dependent on static and heuristic-based data replay strategies. ROAD exhibits robustness across various backbone algorithms and diverse offline data qualities.

Despite the theoretical rigor and promising empirical results, ROAD has limitations that merit discussion. The primary limitation lies in the assumption of linearized mixing strategies and further discretizations of the search space. These assumptions ensure computational feasibility but are essentially projecting our theoretical solution in the infinitedimensional function space G to a low-dimensional subspace, compromising the rich expressivity of the function space. Another limitation stems fromAssumption 1, indicating reliance on the offline dataset. In scenarios with extremely sparse or narrow offline datasets where the Q-value estimation is not sufficiently anchored, the reliability of Rq might degrade.

The promising directions primarily correspond to the limitation of restricted strategy search spaces. One natural extension to ROAD is to incorporate sample-level data selection (as suggested by the wg term in Equation (4)), effectively introducing larger search spaces of mixing strategies. Another future work would be exploring methods to search for the optimal mixing strategy in a continuous space, thus allowing for more delicate control over the training dynamics. Apart from these methodological extensions, the application of this framework to real-world robotic tasks, which emphasizes the algorithm's ability to prevent unlearning while ensuring asymptotic performance, is also of interest.

Acknowledgments

This work was supported by Ant Group Research Fund.

Contribution Statement

The authors Letian Yang and Xu Liu contributed equally.