Post Hoc Extraction of Pareto Fronts for Continuous Control

1 Introduction

Many real-world continuous control problems require agents to balance multiple, even conflicting, objectives. In legged locomotion, for example, a robot must simultaneously optimise forward speed, gait stability, and energy efficiency. In such settings, there is no single optimal solution, and agents must instead discover a set of non-dominated policies that capture the range of feasible trade-offs. Learning this Pareto front enables downstream stakeholders to select behaviours that reflect their current preferences or adapt to changing operational conditions.

Typically, multiple objectives are handled through scalarisation into a single weighted sum and training via standard reinforcement learning (RL). While accessible and tractable, this approach yields only a single, fixed trade-off. Simply repeating this procedure for every trade-off is inefficient and practically infeasible without sophisticated experience and representation sharing.

Multi-Objective RL (MORL), and specifically multi-policy MORL algorithms partially address this through various architectural improvements to learn the entire frontier directly. Methods like MORL/D [Felten et al., 2024] divide the problem into several single-objective problems through scalarisation, and use single-objective RL with buffer sharing to train policies for each scalarisation. Other works like PGMORL [Xu et al., 2020] and MOPDERL [Tran et al., 2023] maintain populations of policies, combining RL with evolutionary principles to jointly improve the policies in the population and cover the objective space. PSL-MORL [Liu et al., 2025], a notable recent work, trains a hypernetwork to produce a fresh policy for every desired trade-off.

Despite successfully learning Pareto fronts, these methods suffer from a critical limitation: requiring full multi-objective consideration from the outset of training. This rigidity creates a disconnect with practical scenarios, where multi-objective preferences may arise retroactively, only after a robust policy for a primary task has already been trained. For instance, a preference for more stability may only emerge after observing a locomotion policy highly optimised for speed. To obtain new trade-offs in these scenarios, practitioners must 1) discard pre-trained policies and incur the sample costs of retraining, and 2) retrofit their algorithm into complex multiobjective learning frameworks. Currently, no method reuses disjoint specialist policies and training data to recover Pareto fronts efficiently and with minimal added algorithmic complexity.

In this work we present Mixed Advantage Pareto Extraction 1 (MAPEX), a novel method that fully leverages priortrained single-objective policies, critics, and replay buffers to produce Pareto fronts of policies. Our key insight is that agents learn optimal trade-offs by intelligently blending expert behaviour on each objective. MAPEX implements this by blending the single-objective evaluation from each specialist critic into a multi-objective mixed advantage value and weighting a behaviour cloning loss with it.

1 Project homepage: https://github.com/raghavthakar/MAPEX

MAPEX bypasses the need to retrofit bespoke or standard off-policy RL into complex multi-objective learning frameworks by training for each objective individually. It preserves both, the simplicity, and intricacies of these algorithms by providing a realistic pathway to learning multi-objective behaviours from single-objective training data. If training from scratch, MAPEX allows allocating the entire sample budget to training high-performing single-objective specialists, from which high-quality Pareto fronts can later be extracted at a minimal sample cost.

We provide a detailed view of MAPEX's Pareto extraction procedure, and perform Pareto extraction from specialists trained using both standard and bespoke off-policy RL. Across five multi-objective MuJoCo environments, MAPEX produces fronts comparable to established baselines from the literature. When extracting Pareto fronts from the same starting policies, other methods consume up to 1000× more samples than MAPEX to produce similar fronts.

2 Background

2.1 Actor-Critic Methods and Offline Reinforcement Learning

Actor-critic methods: These methods perform reinforcement learning using an actor, which defines the policy, and a critic, which estimates the expected return of the actor's behaviour [Sutton and Barto, 2018]. By relying on learned value estimates rather than Monte Carlo returns alone, actor-critic methods enable efficient and stable policy optimisation, and are commonly used in continuous control. These methods are often implemented in an off-policy setting, where experience collected by one or more behaviour policies is stored in a replay buffer and reused for policy updates [Haarnoja et al., 2018; Fujimoto et al., 2018].

Policy improvement is commonly guided by the advantage function,

where Qπ (s, a) is the critic's action-value function, and V π (s) = Ea∼π(·|s)[Qπ (s, a)]. In practice, particularly in deterministic actor-critic methods, the value function is often approximated by Qπ (s, π(s)), yielding

Offline Reinforcement Learning: Offline reinforcement learning studies the problem of learning a policy from a fixed dataset of transitions, without any further interaction with the environment during training. A key challenge in this setting is distributional shift: the dataset may contain diverse experiences that are hard to generalise to, leading to unreliable value estimates [Panaganti et al., 2025; Levine et al., 2020].

Advantage-weighted regression (AWR) [Peng et al., 2019] provides a simple and stable mechanism for policy improvement in this setting by casting reinforcement learning as a weighted supervised learning problem. Given a critic-derived advantage estimate A(s, a), the AWR policy update solves

where β > 0 is a temperature parameter. This objective emphasises actions that are predicted to improve performance while sampled purely from the observed data, making it well suited to off-policy and offline reinforcement learning problems. We use an AWR-inspired regression weighting in MAPEX to learn multi-objective behaviours from static single-objective replay buffers.

2.2 Multi-Objective Sequential Decision-Making

Multi-objective sequential decision-making problems are commonly modelled as Multi-Objective Markov Decision Processes (MOMDPs). A MOMDP is defined by the tuple ⟨S, A, T , γ, R⟩, containing the state space S, the action space A, the transition dynamics T : S ×A× S → [0, 1], and a scalar discount factor γ ∈ [0, 1). Unlike standard MDPs, the reward function R : S × A × S → Rk returns a vector of k rewards, each corresponding to a distinct objective.

A policy π : S → A induces an expected return for each objective. We characterise policy performance directly through its expected long-term returns. Let J(π) ∈ Rk denote the vector of objective returns achieved by policy π, with components

where ri is the reward function associated with the i-th objective and ρ0 is the initial state distribution.

From a multi-objective optimisation perspective, learning in a MOMDP can be viewed as maximising a vector-valued objective

Pareto Dominance and Optimality As objectives are often conflicting, it is generally impossible for a single policy to optimise all objectives simultaneously. Thus, optimality in multi-objective settings is defined in terms of Pareto dominance. Given two objective return vectors v, u ∈ Rk , v Pareto-dominates u (denoted v ≻p u) if v is at least as good as u in all objectives and strictly better in at least one:

If neither vector Pareto-dominates the other, they are said to be non-dominated.

A policy π is Pareto optimal if its return vector J(π) is not Pareto-dominated by that of any other policy. The set of all Pareto-optimal policies induces the Pareto front, which characterises the optimal trade-offs achievable in the objective space.

3 RelatedWorks

Foundational surveys [Hayes et al., 2022; Roijers et al., 2013] categorise the expanding MORL landscape into single-policy and multi-policy approaches. We adopt this taxonomy herein.

Single-policy methods optimise a fixed scalar utility function [Hayes et al., 2022], using standard RL with linear utilities, and dedicated approaches for non-linear utilities [Lin et al., 2024b; Van Moffaert et al., 2013; Reymond et al., 2023]. Conversely, multi-policy approaches approximate the Pareto front, a better-suited approach for unknown or dynamic preferences. Early works like Pareto Q-Learning [Moffaert and Now´e, 2014] tracked non-dominated return vectors for each state-action pair, while recent extensions employ preferenceconditioned deep networks [Abels et al., 2018] or convex envelope updates [Yang et al., 2019] for high-dimensional spaces.

In continuous control, decomposition-based methods divide the problem into several scalarised objectives. Prior work [Chen et al., 2020] has augmented naively solving each scalar objective with an evolutionary strategy [Salimans et al., 2017] for post-processing. Notably, MORL/D [Felten et al., 2024] improves efficiency via cooperative buffer sharing and intelligent weight sampling. Another branch of work combines RL with evolutionary operators: PG-MORL [Xu et al., 2020] iteratively pushes a population of policies toward promising objective space regions, while MOPDERL [Tran et al., 2023] distills a frontier from subpopulations trained via evolutionary RL [Bodnar et al., 2020; Khadka and Tumer, 2018]. Alternatively, parameter-efficient methods consolidate policies using meta-learning [Chen et al., 2019], universal preference-conditioned networks [Basaklar et al., 2023], or hypernetworks [Liu et al., 2025].

Despite architectural differences, these methods share a structural limitation: they are designed solely for "from scratch" learning. If training on one or more individual objectives has already been performed, these methods cannot leverage it, effectively requiring pre-trained policies to be discarded and training to be restarted. The structural rigidity of these approaches also makes adapting bespoke RL algorithms into these frameworks non-trivial.

A separate line of inquiry focuses on offline MORL, training preference-conditioned agents from massive datasets. Approaches like PEDA [Zhu et al., 2023], DiffMORL [Xiao et al., 2025], and PR-MORL [Lin et al., 2024a] utilise transformers, diffusion, and regularisation to generalise across preferences. While seemingly similar, MAPEX tackles the distinct problem of extracting frontiers from pre-trained specialists rather than large-scale generalisation. Consequently, MAPEX operates with replay buffers two orders of magnitude smaller than the D4MORL benchmarks [Zhu et al., 2023] used in offline RL works (e.g., 1 million vs. 150 million transitions).

Finally, PCN [Reymond et al., 2022] also uses supervised learning from an off-policy dataset, but relies on iterative online data collection to populate the objective space. In contrast, MAPEX addresses the strictly offline extraction of frontiers from fixed, disjoint datasets. PCN is also currently limited to discrete action spaces.

4 Method

We now introduce Mixed Advantage Pareto Extraction (MAPEX), an offline algorithm to extract a Pareto front of INPUT: Policy set Π, critic set Q, buffer set D, number of objectives N

Algorithm 1: Mixed Advantage Pareto Extraction (MAPEX)

OUTPUT: Pareto front P of policies 1 FUNCTION MAPEX(Π,Q,D,N): 2 while Required do 3 EVALUATE(Π) 4 P ← FINDPARETOFRONT(Π) 5 {π1, . . . , πN} ← SELECTPARENTS(P) 6 v ← CENTROID(π1, . . . , πN) // In objective space 7 wtarget ← v ∥v∥2 // Target weight vector 8 Dhybrid ← ⋃Nk=1 SAMPLE(Dk, ∝ wk) 9 πnew ← INITPOLICY(∅) 10 for ϵ ← 1 to E do 11 (s, a) ∼ Dhybrid 12 A ← [(Qi(s, a) - Qi (s, πnew(s)))]N i=1 13 Amixed ← w⊤ targetA // Mixed advantage 14 ω(s, a) = min (exp ( Amixed(s,a) β ) , ωmax) 15 LMAPEX ← E[ω(s, a) · ∥a - πnew(s)∥22 ] 16 UPDATE(πnew, LMAPEX) 17 Π ← Π ∪ {πnew} 18 P ← FINDPARETOFRONT(Π) 19 return P

continuous control policies from a set of disjoint singleobjective specialists. The core intent of MAPEX is to fully leverage the latent information in prior-trained specialistswhich includes their policies, along with their value functions (critics) and replay buffers-to produce new policies that express new trade-off behaviours without requiring additional training interaction with the environment.

MAPEX achieves this via a Pareto front extraction procedure which iteratively fills gaps in the Pareto front estimate. First, MAPEX analyses the available policies in the objective space to identify sparse regions in the Pareto front estimate. Once a gap is identified, MAPEX derives a vector of 'target weights' that encodes a weighting over the objectives that would optimally fill this gap. To train a policy for this new trade-off, MAPEX constructs a static hybrid buffer by sampling from the specialists' buffers in proportion to these target weights. Crucially, the algorithm then calculates a mixed advantage for every transition in this dataset by querying each specialist critic, and mixing the individual advantage estimates in the ratio of the target weights. This mixed advantage captures the value of a transition in demonstrating target trade-off behaviour. Finally, a fresh policy is trained using a mechanism inspired by Advantage Weighted Regression (AWR) [Peng et al., 2019], wherein we regress the policy onto actions from the hybrid buffer, weighted by an exponential of their calculated mixed advantage. Algorithm 1 provides a more rigorous look at this procedure.

Starting Information and Notation

We assume that from prior training we have a policy set Π with at least N policies for an N-objective problem, a critic set Q with N critics, each specialising on evaluating on one of the problem's objectives, and a replay buffer set D with N buffers, each containing experiences from training on a single objective.

4.1 Step 1: Gap Identification and Parent Selection

At the start of each iteration, we evaluate the current policy set Π to obtain the performance vector J(π) ∈ RN of each policy. We then identify the non-dominated set (the current Pareto front approximation). We then search for the largest N-dimensional 'gap' on the frontier-a sparse region in the objective space. For N = 2, we define this as the edge with the maximal Euclidean span. In practice, to avoid repeatedly focusing on the same gap, we select the gap using roulettewheel sampling based on edge length. The N policies corresponding to the vertices of this gap are selected as the parent policies, denoted by {πp1 , . . . , πpN }.

To guide the offspring into this sparse region, we compute the centroid of the parents' performance vectors in the objective space:

We then derive a unit vector wtarget of target weights pointing towards this centroid. This vector encodes the linear preference required to interpolate the gap and guides the subsequent hybrid buffer creation and advantage mixing steps.

4.2 Step 2: Hybrid Buffer Creation and Advantage Mixing

After deriving the target weight vector wtarget, we assemble a training distribution that reflects this desired trade-off. We construct a fixed-size hybrid buffer, Dhybrid, by sampling transitions from each specialist's buffer Dk in direct proportion to the corresponding weight wtarget,k. This creates a dataset that is structurally biased to each objective in the desired proportion.

We then initialise a random policy network πnew that will be optimised to achieve the target trade-off behaviour. For this optimisation, we iteratively sample transitions from Dhybrid to compute the mixed advantage training signal. For each transition (s, a), we compute a vector of advantages A(s, a) ∈ RN . The kth element of this advantage vector is the advantage on the kth objective associated with that transition, and is computed using the kth specialist critic Qk:

This formulation leverages the specific value estimation expertise of each critic for their respective objective.

Finally, we scalarise these vector-valued advantages into a single training signal. We compute the mixed advantage as the dot product of the advantage vector and the target weights derived in Step 1:

This scalar value Amixed represents the quality of the state-action (s, a) specifically regarding the desired trade-off wtarget .

4.3 Step 3: Mixed AdvantageWeighted Regression

To train the offspring policy πnew, we employ a supervised regression objective weighted by the scalarised signal derived in Equation 9. Our goal is to selectively clone actions that contribute positively to the specific target trade-off wtarget.

For a transition (s, a) we compute a regression weight ω(s, a) by applying a temperature-scaled exponential to its mixed advantage:

where β > 0 is a temperature hyperparameter and ωmax is a clipping threshold to ensure numerical stability.

The fresh policy πnew is then optimised to minimise the weighted mean squared error between its predicted action and the retrieved buffer action a:

Once the policy has been updated for the desired epochs, it is reinserted into the population Π and the MAPEX procedure is executed for another iteration.

4.4 Mitigating Out-of-Distribution Error

While MAPEX's mixed advantage values are intuitive to compute, they expose two main sources of out-of-distribution (OOD) error: 1) when a transition sampled by one objective's specialist policy is evaluated by another objective's specialist critic, and 2) when a randomly-initialised policy's action is evaluated by specialist critics. We mitigate these issues using the following design choices.

Secondary Critics

When training each specialist policy πk, we learn not only its primary critic for objective k, but also a set of secondary critics for the remaining objectives. All critics associated with specialist k are trained on the same replay buffer Dk generated by πk, so that every objective can be evaluated on data that is in-distribution for the corresponding critic.

Notation: Let m ∈ {1, . . . ,N} index objectives and k ∈ {1, . . . ,N} index specialists. We denote by Q(k) m (s, a) the action-value critic that predicts the return for objective m using transitions sampled from Dk (i.e., collected under πk). Under this notation, the specialist's primary critic is Q(k) k , while the secondary critics are {Q(k) m }m̸=k.

Procedure: During specialist training, only the primary critic Q(k) k is used to update the policy πk. The secondary critics {Q(k) m }m̸=k are trained in parallel on the same buffer Dk, but they do not contribute gradients to the policy update. After training all specialists, we collect the resulting critics into a single familyQ ≜ {Q(k) m }k=1...N, m=1...N. This construction ensures that when MAPEX samples a transition (s, a) that originates from a particular specialist buffer Dk, the evaluations {Q(k) m (s, a)}Nm=1 are produced by critics trained on the same state-action distribution as the sampled data.

Practical note: Secondary critics may be trained alongside primary critics with no change to the policy update rule. If single-objective specialists have already been trained, a secondary critic {Q(k) m }m̸=k can be trained offline using transitions from Dk and the corresponding objective rewards. Either way, a distribution-matched critic family Q can be learnt for a low cost. We include a comparison of joint-vs. post-hoctrained secondary critics in Section 5.

Offspring Policy Warm-Up

In step 2 of the MAPEX procedure (Equation 8), computing the mixed advantage requires querying specialist critics with actions proposed by πnew(s). If πnew is initialised arbitrarily, these actions can be far from the support of the hybrid buffer and cause OOD error. To reduce this effect, we warm up πnew by regressing it to the mean of its parents in the action space.

Let {πp1 , . . . , πpN } denote the parent policies. The mean parent action at state s is ¯a(s) ≜ 1N ∑Nj=1 πpj (s), which we use to perform a brief behavioural regression step that minimises

We run this regression for a small number of gradient steps prior to computing the advantage vector in Equation 8. This warm up keeps πnew's predicted actions close to those produced by the parents on states drawn from Dhybrid, more closely matching each critic's training exposure, and improving the reliability of subsequent critic-based updates.

5 Experiments

We evaluate MAPEX on three criteria: sample efficiency of Pareto extraction; flexibility regarding choice of specialist training algorithm and nature of secondary critic training (joint vs. post-hoc); and general competitiveness against baselines when training from scratch.

We assume access to secondary critics trained alongside primary critics during specialist training, but also test a post hoc variant (MAPEX-PostHoc) where critics are trained offline on static buffers. Unless stated otherwise, specialists are trained using Proximal Distilled Evolutionary RL (PDERL) [Bodnar et al., 2020].

5.1 Baselines and Domains

We compare MAPEX against two established MORL methods from the literature:

• MOPDERL [Tran et al., 2023]: An evolutionary actorcritic method that first trains on each individual objective using PDERL, and later crosses over solutions across objectives using a multi-objective distilled crossover. It serves as a baseline for both 'from scratch' training and pure Pareto extraction (via its distillation phase).
• MORL/D [Felten et al., 2024]: A decomposition-based approach that uses Soft Actor-Critic [Haarnoja et al., 2018] on each scalar objective. It uses buffer data sharing and adapts the scalar weights using Pareto Simulated Annealing [Czyz˙zak and Jaszkiewicz, 1998].

We use the morl-baselines [Felten et al., 2023] implementation of MORL/D and the authors' original implementation of MOPDERL 2 with default hyperparameters. Experiments are conducted on five bi-objective continuous control MuJoCo environments from MO-Gymnasium [Felten et al., 2023], with episodes capped at 750 frames3 .

5.2 Experimental Setup

Sample efficiency: We compare the extraction phase of MAPEX against the distillation phase of MOPDERL starting from identical specialist subpopulations trained with PDERL. Flexibility: We compare standard MAPEX against MAPEXPostHoc and MAPEX-TD3 (specialists trained via TD3) to assess robustness to starting policies and nature of critic training. Competitiveness: We compare the final Pareto fronts of the full MAPEX pipeline against MOPDERL and MORL/D given a fixed sample budget. During specialist training of all MAPEX variants, we use a replay buffer of size 1M. In each test we perform five seeded runs with each method 3 .

6 Results

6.1 Sample Efficiency of Extraction

MAPEX achieves a massive reduction in sample cost compared to baselines. As shown in Figure 1 (MO-Ant-v5), MAPEX and MAPEX-PostHoc extract a high-performing front almost immediately (requiring only evaluation rollouts), while MOPDERL requires an additional 300, 000 environment interactions to attain the same performance.

Figure 2 confirms this trend across MO-Hopper-v5 and MO-Walker2d-v5. Particularly in MO-Hopper-v5, MAPEX requires 100 samples to reach hypervolume thresholds that MOPDERL requires ≈ 105 samples to attain-a reduction of three orders of magnitude. This efficiency is driven by MAPEX exploiting the latent representations of multiobjective behaviour in policies and replay buffers. It empirically validates our intuition that following expert behaviour to varying degrees on each objective yields trade-offs in the objective space. It also validates our main hypothesis that these behaviours can be learnt by regressing onto target actions, weighed by their mixed advantage value.

2 We made minor modifications to the original MOPDERL code to allow testing it on v5 MO-Gymnasium MuJoCo domains. Code: https://github.com/raghavthakar/MOPDERL-MO-Gymnasium

3 Exact details of the environment, with algorithm-specific and experimental parameters are available on the project homepage: https://github.com/raghavthakar/MAPEX

6.2 Flexibility and Robustness

MAPEX is robust to the source of specialist policies. Figure 3 shows that Pareto extraction for standard MAPEX, MAPEXPostHoc, and MAPEX-TD3 are largely indistinguishable on MO-Walker2d-v5 and MO-Ant-v5. Crucially, the success of MAPEX-PostHoc confirms that secondary critics can be effectively trained retroactively on static buffers, enabling Pareto extraction from fully decoupled single-objective training. If simplicity is key, then the performance of MAPEXTD3 shows the potential of elegantly adapting off-policy RL

to learn multi-objective behaviours via MAPEX.

While a population of policies (like that produced by PDERL) provides a rich starting point for Pareto extraction, MAPEX-TD3 remains competitive with only one specialist policy per objective (as produced by simple TD3). This is because MAPEX does not simply distil or interpolate between parent policies. Instead, MAPEX draws expertise from the pre-trained replay buffers, which contain a rich and varied set of state-action examples. These experiences span regions of the objective space that an individual policy may only cover sparsely, allowing MAPEX to produce distinct policies for distinct trade-offs. This is also why MAPEX is robust to the exact algorithm used to train specialist policies.

6.3 General Competitiveness

Despite being an offline extraction method, MAPEX produces fronts competitive with MOPDERL and MORL/D, which require full multi-objective consideration from the beginning. Table 1 shows that MAPEX achieves comparable hypervolumes (e.g., 3.34 × 106 vs. MOPDERL's 3.17 × 106 in MO-Hopper-v5, and 1.78×105 vs. MORL/D's 9.29×104 in MO-Swimmer-v5) across five multi-objective MuJoCo environments. While MAPEX fronts are numerically sparser, by inspecting the fronts in Figure 4 visually, it is clear that MAPEX produces even and well-spread Pareto fronts.

7 Conclusion

We presented MAPEX, a novel approach to extracting Pareto fronts of policies from prior single-objective training for continuous control. We provided a detailed view of MAPEX's Pareto extraction procedure, and mentioned some practical implementation tips. We tested MAPEX with wellestablished, dedicated MORL baselines like MOPDERL and MORL/D to empirically validate its mixed advantage approach. Pareto fronts are learnt cheaply with MAPEX when specialists are already trained. If training from scratch, MAPEX integrates easily with off-policy RL methods and still produces fronts that compare well to baselines. Finally, we discuss some limitations and future work.

While MAPEX is highly sample efficient, it makes assumptions inherent to our offline extraction setting. First, MAPEX is strictly bounded by the support of the specialist buffers; it cannot discover novel behaviours or skills that are absent from the specialists' training history. Second, MAPEX relies on the assumption that valid trade-off policies lie on a continuous manifold between specialists. In scenarios where specialists exhibit markedly disjoint behaviours (e.g., a humanoid walking on two legs vs. crawling), interpolation may yield low-performance policies. Third, when a new objective arises, rewards on that objective may not exist in the collected data, necessitating post-hoc reward reconstruction over the replay buffer using logged telemetry data or fresh rollouts.

Our empirical evaluation focused on bi-objective domains; scaling MAPEX's gap-identification heuristic to more objectives (N ≥ 3) remains a subject for future work. We also aim to leverage MAPEX in the multi-objective multiagent setting, integrating with reinforcement learning [Foerster et al., 2018] and credit assignment methods [Thakar et al., 2025].

Acknowledgements

This work was supported by the Office of Naval Research grant N00014-23-1-2171. We thank Christopher M. Sullivan at Oregon State University for generously providing access to high-performance computing resources. We also thank the reviewers for their precise and useful feedback on our work.