EVA-Gen: When Perception Learns from Value via Generative Models in Decentralized Multi-Agent Systems

1 Introduction

The field of Multi-Agent Reinforcement Learning (MARL) [Zhang et al., 2021; Bus¸oniu et al., 2008] has rapidly become a cornerstone for distributed control in complex systems where centralized sensing and decision-making are impractical. These domains heavily rely on agents that must make decisions based on limited local observations and con-

* Corresponding author.

（a）Standard Decoupled

Perception-Decision Pipeline

（b）Value-Conditioned

Cybernetic Loop（EVA-Gen）

Figure 1: Overview: (a) The standard decoupled pipeline, where perception minimizes uniform reconstruction loss independent of decision relevance. (b) EVA-Gen, which closes the perceptioncontrol loop by steering generative belief inference toward valuesen·sitive regions via Backward Flow.

strained communication channels. In these decentralized execution environments, each agent operates with a locally consistent yet globally incomplete view of the system, a challenge often referred to as epistemic fragmentation [Oliehoek and Amato, 2016; Dibangoye and Buffet, 2018]. Further compounding this issue is the presence of heterogeneous value landscapes, where agents must navigate through regions that carry disproportionately high stakes due to sparse, critical events. These challenges make real-world MARL deployment notoriously brittle.

Traditional model-based architectures often decouple state representation from policy optimization. In these frameworks, perception modules are trained to minimize a uniform reconstruction error across the observation space, treating background noise and critical signals equally. While notable works such as DreamerV3 [Hafner et al., 2025] and MuZero [Schrittwieser et al., 2020] have made strides toward value-aware learning, they generally target settings with either single-agent or fully observable environments where a unified world model suffices. In decentralized systems, this separation creates a fundamental pathology: a PerceptionValue Misalignment, where agents waste scarce computational resources processing irrelevant background textures while failing to resolve uncertainties in regions where the value function gradient is most sensitive. This misalignment is especially detrimental when operating under tight communication or compute budgets.

To resolve this misalignment, we propose a paradigm shift that places the value function at the center of perception learning (Fig. 1). Our insight is rooted in the observation that under resource constraints, perceptual errors do not equally impact decision-making. Specifically, if an agent's objective is to maximize the expected return V π (s), the impact of a perceptual error ε = ˆ s - s can be approximated by a first-order Taylor expansion: V π (ˆ s) ≈ V π (s) +∇sV π (s)⊤ε. This relationship reveals that the error's impact on utility is proportional to the gradient magnitude ∥∇sV π (s)∥. This observation motivates the concept of Value-Conditioned Reconstruction: optimal perception under fixed resource constraints is not uniform, but functionally weighted by the value gradient, prioritizing fidelity in high-stakes regions.

In response, we introduce EVA-Gen (Epistemic-Value Alignment via Generative Models), a unified framework that bridges the gap between generative perception and valuebased decision-making. EVA-Gen incorporates this principle through three synergistic mechanisms: (1) Backward Flow via Value-Guided Diffusion: We utilize diffusion models [Ho et al., 2020; Janner et al., 2022] to model the multimodal uncertainty in fragmented observations. By introducing a value-gradient-weighted sampling process, we bias the reverse diffusion steps to prioritize the generation of highvalue latent manifolds, effectively guiding perception toward task-relevant areas. (2) Collaborative Information Bottleneck: To mitigate decentralized bandwidth limitations, we introduce a collaborative information bottleneck [Wang et al., 2020] that filters communication. By maximizing mutual information specifically over value-relevant features, this mechanism fosters consensus on critical perceptual elements while discarding shared noise. (3) Risk-Sensitive Gradient Rectification: In environments with sparse high-value events, we counter signal dilution by adjusting policy gradients using uncertainty estimates derived from the generative model. This ensures the policy remains sensitive to sparse but critical signals despite the prevalence of low-value data.

In summary, the contributions of this work are as follows:

1. We formulate the Value-Conditioned Reconstruction Paradigm, establishing that optimal perception under resource constraints should be weighted by the value function gradient.
2. We introduce EVA-Gen, a novel architecture that integrates value-guided diffusion, collaborative bottlenecks, and risk-sensitive rectification to achieve tight epistemic-value alignment.
3. We provide extensive empirical validation on SMAC variants with sparse rewards, Value-oriented collaborative exploration tasks, and an edge computing testbed.

2 RelatedWork

Our work positions EVA-Gen at the intersection of generative belief inference, value-aware modeling, and communicationefficient coordination under resource constraints.

2.1 Generative beliefs under epistemic uncertainty

Decentralized control relies on inferring global states from partial observations. While recurrent encoders paired with CTDE methods (e.g., QMIX [Rashid et al., 2020]) provide scalable coordination, they typically yield deterministic pseudo-beliefs that do not explicitly represent the multimodal posteriors induced by severe occlusion [Wang et al., 2023]. This motivates generative belief modeling in POMDP RL [Igl et al., 2018], where diffusion models improve mode coverage [Ho et al., 2020; Janner et al., 2022]. However, existing generative perception approaches optimize uniform denoising or reconstruction objectives, inducing a frequency bias: modeling capacity is allocated by visitation density rather than decision sensitivity. EVA-Gen bridges this gap by conditioning diffusion emphasis on value gradients, ensuring limited compute is spent on resolving high-stakes uncertainty.

2.2 Value equivalence vs. value conditioned reconstruction

Model-based RL (e.g., Dreamer, MuZero) learns latent dynamics for planning and control [Schrittwieser et al., 2020]. A key principle is Value Equivalence [Grimm et al., 2020]: representations need only preserve information relevant to Bellman updates. While conceptually aligned with task relevance, these methods are typically realized in centralized settings with objectives that remain effectively frequencyweighted; in heterogeneous value landscapes, this systematically underserves rare, high-impact regions. EVA-Gen complements value-equivalent modeling by shifting the focus from state abstraction to resource allocation, prioritizing perceptual fidelity according to value sensitivity under explicit compute/bandwidth budgets.

2.3 Communication-efficient coordination and risk-sensitive learning

To handle bandwidth constraints, explicit communication methods learn what to share [Sukhbaatar et al., 2016], while Information Bottleneck objectives motivate compression [Alemi et al., 2017]. Yet multi-agent bottlenecks define relevance via generic predictability [Su et al., 2024], which may preserve shared but low-utility features. EVA-Gen introduces a Collaborative Information Bottleneck shaped by value-relevant supervision, so transmitted bits reduce uncertainty along decision-critical dimensions. While prioritization (e.g., PER [Bellemare et al., 2017]) can amplify rare signals, EVA-Gen couples such emphasis with closed-form importance correction and uncertainty-aware rectification to preserve gradient fidelity under value-biased perception.

3 Preliminary

We begin by defining the resource-constrained decentralized control problem under epistemic fragmentation, then show why uniform perception is theoretically misaligned with control via a first-order value-degradation analysis.

3.1 ProblemFormulation

We consider a Resource-Constrained Decentralized Partially Observable Markov Decision Process (RC-Dec-POMDP),

where N = {1, . . . ,N} is the agent set. At time t, the environment is in global state st ∈ S. Each agent i receives a local observation

which is severely occluded/noisy, inducing epistemic fragmentation. Agents take actions ai,t ∈ Ai, forming a joint action at = (a1,t, . . . , aN,t). The system transitions as st+1 ∼ P(· | st, at) and yields a shared reward rt = R(st, at).

Because st is unavailable at execution, EVA-Gen operates on a control-sufficient latent state zt ∈ Rd , defined by an encoder φ (learned under CTDE supervision):

During execution, agent i infers a belief ˆ zi,t from its local observation and received messages (defined below), and executes a decentralized policy

We use CTDE to train a centralized critic on zt, while enforcing decentralized execution through Eq. (4).

Agents can exchange compressed messages. Let ui,t be agent i's transmitted message and ˜ui,t the received (possibly corrupted/dropped) message. We impose the constraint set

where Bandwidth constraint: ∑Ni=1 bits(ui,t) ≤ B for each timestep t. Compute constraint: the perception module qϕ has a bounded inference budget, e.g., a maximum number of denoising steps or FLOPs Ccomp.

Let ci,t denote the aggregated communication context available to agent i (defined in Sec. 4.2). The generative perception module produces a posterior over latent belief:

The goal is to maximize the expected discounted return:

Under these constraints, perception cannot be uniformly accurate everywhere; the core question becomes how to allocate limited perceptual capacity to maximize control utility.

3.2 Theoretical Motivation

Standard model-based MARL pipelines decouple perception from control, training perception to minimize a uniform reconstruction loss (e.g., E∥z - ˆ z∥2 ).

Let V π (z) be the value function of policy π in the latent space, and define the perception error ε ≜ ˆ z - z. We denote the (discounted) visitation distribution under π by ρπ (z).

Theorem 1 (Value-Conditioned Reconstruction Principle). Assume V π is differentiable and L-smooth on the support of ρπ , i.e., ∥∇V π (z) -∇V π (z′ )∥ ≤ L∥z - z ′ ∥. Then for any estimator ˆ z,

Note: Setup indicates accessibility (✓: door open, ◦: partially open, ×: closed). CRB (Clean); PFS/CAT/VMD (Single stressor); GUD (Gated+σ+C); TCB (All factors coupled).

Moreover, for any δ0 > 0,

which implies that, under a fixed distortion budget E∥ε∥2 ≤ δ, minimizing the value-conditioned quadratic objective

minimizes a tight upper bound on the expected value degradation in Eq. (8) up to constants.

Proof. By L-smoothness, Taylor's theorem gives V π (ˆ z) = V π (z) + ∇V π (z)⊤ε + R with |R| ≤ L2 ∥ε∥2 , which yields Eq. (8) by Cauchy-Schwarz. Eq. (9) follows from the inequality ∥x∥ ≤ 1 2δ0 ∥x∥2 + δ0 2 applied to x = ε and scaling by ∥∇V π (z)∥. Substituting into Eq. (8) shows that LVCR controls the leading term governing value deviation under a quadratic distortion budget.

Remark 1 Theorem 1 establishes a precise surrogate necessity: under finite resources, perception should prioritize regions where the value function is sensitive (large ∥∇zV π (z)∥). Uniform reconstruction is recovered only in the degenerate case where ∥∇zV π (z)∥ is constant (i.e., no value heterogeneity).

4 Methodology

Guided by Theorem 1, EVA-Gen Framework closes the perception-value loop via three mechanisms (Fig. 2). We utilize a CTDE-trained centralized critic Vω(z) to compute value sensitivities in the latent space.

4.1 Backward Flow via Value-Guided Diffusion

To minimize Eq. (10) under compute constraints, we allocate capacity to value-sensitive latents and implement Backward Flow via value-weighted diffusion.

Value-weighted replay distribution: define the value sensitivity and sampling weight

where α ≥ 0 controls aggressiveness and stabilizes training.

Value-weighted replay distribution: given a replay buffer distribution pD(z, o, c), construct a value-biased distribution

Sampling from pval implements the value-conditioning in Eq. (10) while retaining the same model class.

Diffusion training in latent space: we train a conditional diffusion model ϵϕ to model qϕ(z | o, c) by predicting injected noise in the standard denoising objective [Ho et al., 2020; Song et al., 2021b]. Let zk denote the noised latent at diffusion step k. The Backward Flow objective is

Note: Setup indicates degradation tier (✓: single mild factor, ◦: two-factor coupling, ×: full stress). CBC (Clean); OCC/NOI/DEC (Single stressor); DPD (Dual perception degradation); FULL (Max stress).

To satisfy Ccomp, we use accelerated samplers (e.g., DDIM [Song et al., 2021a]) with K′ ≪ K denoising steps at execution. Backward Flow ensures these limited steps focus on resolving uncertainty in high-stakes manifolds.

4.2 Collaborative Information Bottleneck (CIB)

Under bandwidth budget B, agents cannot share raw observations. CIB learns compressed messages that retain valuerelevant information and align agents on critical factors.

CIB encoder and aggregation (interface to diffusion): each agent encodes a message

aggregates received neighbor messages into a context vector

which is provided to diffusion as the conditioning input in Eq. (6), Eq. (15). This couples communication to perception.

Information bottleneck objective: let yt be a value-relevant training signal, e.g., yt = Vω(zt), an advantage estimate ˆ At , or a binary criticality label I{g(zt) ≥ τ}. We seek messages that preserve information about yt while compressing oi,t:

where β trades off informativeness and rate. In practice, we optimize a standard variational information bottleneck surrogate [Alemi et al., 2017]:

where p(u) is a simple prior (e.g., N(0, I)) and the KL term acts as a differentiable proxy for expected bitrate.

4.3 Risk-Sensitive Gradient Rectification

Value-Guided Diffusion biases samples toward highsensitivity states, which is necessary for perception but can distort policy learning if uncorrected. Meanwhile, sparse critical events risk being overwhelmed by routine transitions.

Proposition 1 (Gradient dilution under value sparsity). Let Zcrit and Zrtn be critical and routine sets with pcrit = P(z ∈ Zcrit) and prtn = 1 - pcrit. Decompose the expected policy gradient as G = pcritGcrit+prtnGrtn. If prtn/pcrit = κ ≫ 1 and Gcrit,Grtn are comparable in magnitude, then under uniform sampling the critical-state contribution is O(1/κ).

Because pval(z, o, c) ∝ pD(z, o, c) w(z) (Eq. (13)), density ratio required to correct bias is available in closed form:

Thus, we do not perform density-ratio estimation; we directly use 1/w(z) (with clipping/normalization) as the biascorrection factor.

Diffusion defines a posterior distribution over ˆ zi,t. Quantify epistemic uncertainty by Monte Carlo posterior spread:

In practice, M is small (e.g., M ∈ {2, 4}) and can be computed only during training or amortized at execution.

We define the rectification weight

where ¯σdiff,t aggregates per-agent uncertainties (e.g., mean over i) and λ > 0 controls risk sensitivity. The rectified actor update is then

with an analogous ρt-weighted critic regression. The biascorrection restores gradient fidelity under 4.1's value-biased sampling, while the uncertainty gate suppresses unstable updates under highly ambiguous beliefs.

5 Experiments

We conduct comprehensive experiments using the EVAGen framework across three scenarios characterized by epistemic fragmentation and heterogeneous value landscapes: (i) MACE-GridWorld [Jiang et al., 2024] for fundamental mechanism verification, (ii) SMAC for complex combat coordination [Samvelyan et al., 2019], and (iii) a custom UAVMEC testbed for system-level validation.

5.1 Baselines

We compare EVA-Gen with five baselines: IPPO [De Witt et al., 2020], an independent on-policy learner without explicit coordination; MAPPO [Yu et al., 2022], a strong CTDE PPO baseline with centralized value estimation and decentralized execution; MACE [Jiang et al., 2024], using decentralized novelty sharing for coordinated exploration; CMiMC [Li et al., 2024], maximizing mutual information between local and collaborative views for communication-efficient MARL; and ExplabOff [Ren et al., 2025], an MI-enhanced collaborative offloading baseline. All methods follow the same training protocol and network scale for fair comparison.

5.2 Experimental Environments

GridWorld: We adopt GridWorld benchmarks, Pass, SecretRoom, and MultiRoom, for sequential critical state reasoning. We introduce an Epistemic-Value Trap via perturbations: (1) Positional Drift (σ) for epistemic fragmentation via stochastic coordinate shifts; (2) Ambiguity (D) for perceptual aliasing using dummy triggers; and (3) Distractors (C) for value heterogeneity via dense low-value rewards.

SMAC: Experiments are conducted on SMAC (2m-1z, 3m, 8m) under sparse rewards [Jiang et al., 2024]. We augment the standard environment with three stressors to test robustness: (i) Dynamic Occlusion (stochastic feature masking); (ii) Observation Noise (attribute perturbation); and (iii) Value Decoys (dense, low-value targets), which create value traps to distract agents from global coordination.

UAV-MECSystemVerification: In our customUAV-MEC simulation, tasks exhibit heterogeneous QoS requirements: a small fraction of high-priority, latency-sensitive tasks (LS, high value) and a majority of best-effort tasks (LT, low value). Agents operate under partial observability due to channel occlusion and incomplete communications.

5.3 Experimental Analysis

GridWorld Results

Table 1 shows Success Rate (SR) and Episodic Return (ER) in scenarios under six configurations of epistemic fragmentation and value heterogeneity. The near-perfect performance

Note: GridWorld: Door=closed, σ = 0, Dummy=low, Coins=50; SMAC: σ = D = 0.1; UAV-MEC: obs.-noise.

of the clean baseline (CRB) confirms task reachability. EVAGen excels: in PassRoom, it lifts SR from 52.1% (IPPO) to 70.7%; in MULTIROOM, it improves from46.2% (MACE) to 68.1%. While value-agnostic baselines show high variance in difficult regimes-indicating policy divergence-EVA-Gen maintains tight error margins (±2 ∼ 3%). Higher ER indicates that EVA-Gen succeeds efficiently without time penalties. This validates that conditioning perception on downstream value gradients suppresses task-irrelevant noise, resolving perception-value misalignment.

SMAC Results

Table 2 reports WR and ER on three SMAC maps under six stress settings with increasing perceptual degradation and value heterogeneity. EVA-Gen achieves the strongest robustness across maps, improving the average WR on 2m vs 1z from 63.9% (MACE) to 71.8% and maintaining 68.5% WR under the FULL setting. On the harder 8m map, EVA-Gen obtains the highest average WR (12.9%) and ER (278.2), outperforming MACE (10.7%WR, 255.5 ER). As stress increases, baselines degrade more sharply, whereas EVA-Gen preserves higher WR/ER, indicating more reliable coordination under severe partial observability and value decoys. These results support that value-guided perception mitigates perception-value misalignment in sparse-reward MARL.

UAV-MEC Results

Table 3 presents UAV-MEC performance across Urban, Emergency, and Dense scenarios under varying stress (noise, packet loss, QoS criticality). The results demonstrate a critical joint improvement: EVA-Gen simultaneously increases SR while reducing the LS Task Violation Rate (VR). Specifically, in Urban scenarios, it attains 86.09% SR with only 13.91% VR, substantially outperforming MAPPO and strong offloading baselines like CMiMC. Unlike aggressive strategies that trade safety for marginal success gains, EVA-Gen pushes the Pareto frontier outward, indicating superior risksensitive trade-offs under uncertainty. This supports our argument: by aligning reconstruction resources with value and risk, the system prevents sparse high-stakes signals from being diluted by abundant low-value channel fluctuations, ensuring reliability where it matters most.

Ablation Study Results

Table 4 confirms the synergy of EVA-Gen, which consistently outperforms its ablated variants in performance (SR/WR), reliability (VRLS/ER), and efficiency (AUC). w/o BF: Removing Backward Flow reduces AUC and success rates, especially under severe uncertainty such as UAV-MEC, showing that value-guided belief reconstruction is crucial for stable convergence. w/o CIB: Without CIB, agents transmit value-irrelevant signals, leading to higher constraint violations and weaker coordination under limited communication. w/o RSR: Removing RSR sharply increases V RLS in UAVMEC despite moderate success rates, revealing a value trap: agents overfit frequent low-value rewards while neglecting sparse high-priority constraints. This validates RSR's role in preserving reliability under long-tailed value distributions.

6 Conclusion

In this work, we proposed EVA-Gen, a unified decentralized MARL framework for mitigating Perception-Value Misalignment under epistemic fragmentation and heterogeneous value landscapes. We established the Value-Conditioned Reconstruction Principle, showing that value degradation induces a perception objective weighted by value sensitivity, and instantiated it through a closed loop of Backward Flow, Collaborative Information Bottleneck, and Risk-Sensitive Gradient Rectification. Our results suggest that resource-constrained agents should prioritize decision-critical uncertainty over uniform world reconstruction.

Acknowledgements

This work was supported in part by the Artificial Intelligence Scientific and Technological Innovation Program of Liaoning Province (No.2023JH26/10100008)

Contribution Statement

YuDuo Zheng and XueFeng Du contributed equally to the formulation, method design, experiments, analysis, and writing of this work. Yanqi Cheng supported the implementation and experimental analysis. Li Yin contributed to validation and revision. Fengqi Li supervised the project, revised the manuscript, and serves as the corresponding author. All authors have read and approved the final manuscript.