Phi-Actor-Critic: Steering General-Sum Games to Pareto-Efficient Correlated Equilibria

1 Introduction

Achieving socially efficient coordination, where agents collectively obtain higher welfare, remains challenging in multiagent systems with partially conflicting incentives. Even when higher-welfare collective strategies exist, independent learning dynamics in mixed-motive environments often converge to individually rational but socially inefficient behaviors, such as risk-averse coordination or persistent asymmetric roles [Shoham and Leyton-Brown, 2008]. Existing deep multiagent reinforcement learning (MARL) methods often struggle to avoid such outcomes. Value-decomposition approaches such as QMIX [Rashid et al., 2020] rely on restrictive monotonicity assumptions, while policy-gradient methods (PGMs) such as MAPPO [Yu et al., 2022] lack mechanisms that explicitly select socially desirable equilibria. As a result, these methods may converge to stable but low-welfare Nash equilibria (NE).

To address this limitation, we shift the focus from standard reward maximization to regret-based equilibrium selection rooted in algorithmic game theory (AGT) [Hart and MasColell, 2000; Blum and Mansour, 2007; Cesa-Bianchi and Lugosi, 2006]. Rather than targeting only independent NE, regret-based learning enables correlated solution concepts that support richer coordination in general-sum games. In particular, while external regret minimization is associated with broad coarse correlated equilibria (CCE), swap-regret-based learning refines the target to the tighter class of correlated equilibria (CE) by evaluating whether an agent would benefit from conditionally replacing one action with another [Hart and Mas-Colell, 2000; Blum and Mansour, 2007].

Based on this insight, we propose Φ-Actor-Critic (Φ-AC), a regret-aware framework designed to operationalize swap regret for equilibrium selection in deep MARL. We address two practical challenges in applying swap regret to deep MARL: scalable regret estimation and equilibrium selection. First, since calculating swap regret typically requires computationally expensive counterfactual evaluation, we introduce a centralized attention critic that efficiently predicts vector-valued regrets in a single forward pass (O(1)). Second, since low regret only encourages behavior consistent with approximate CE conditions, it does not specify which equilibrium within the CE set should be selected. We therefore introduce the Regret-Balancing Social Welfare Objective (RB-SWO), which biases learning toward Pareto-efficient CE by optimizing social welfare under regret constraints. By formulating the learning problem as a constrained Markov game optimized via a Lagrangian objective, Φ-AC encourages socially efficient behavior while maintaining low swap regret.

Our contributions are threefold:

1. Efficient Regret Estimation. We introduce a regretconditioned critic that predicts swap regret in a single forward pass, avoiding expensive counterfactual simulations in high-dimensional environments.
2. Principled Equilibrium Selection. We introduce a

Lagrangian-based selection mechanism that steers learning toward efficient and fair equilibria under regret constraints.

1. Scalable Coordination in Social Dilemmas. We demonstrate that Φ-AC learns high-welfare coordination strategies in matrix games and scales to complex Sequential Social Dilemmas (SSDs), outperforming strong MARL baselines in both sustainability and fairness.

2 RelatedWork

Scalable Deep MARL. Centralized-training decentralizedexecution (CTDE) has become a standard paradigm for scalable MARL, allowing methods such as MADDPG [Lowe et al., 2017] and COMA [Foerster et al., 2018] to use centralized critics during training while preserving decentralized execution. Within this paradigm, COMA improves credit assignment through counterfactual baselines, while value-decomposition methods such as QMIX [Rashid et al., 2020] further improve scalability by factorizing the joint action-value function through a monotonic mixing assumption. Recent PGM such as MAPPO [Yu et al., 2022] also demonstrate strong empirical performance across cooperative MARL benchmarks. Despite these advances, these methods remain primarily optimized for expected return. QMIX is further constrained by its monotonic mixing assumption, which can be restrictive in mixed-motive games, while PGMs do not explicitly impose regret-based equilibrium selection. Thus, they may achieve stable learning or high returns without controlling whether the selected outcome is socially desirable.

Regret-Based Learning. Regret-minimization methods provide a principled route to equilibrium computation. Counterfactual Regret Minimization (CFR) and its neural variants, including Deep CFR and DREAM, minimize counterfactual regret and provide convergence guarantees primarily in twoplayer zero-sum imperfect-information games [Zinkevich et al., 2007; Brown et al., 2019; Steinberger et al., 2020]. Similarly, NeuRD connects policy-gradient learning to Hedge [Freund and Schapire, 1997] and replicator dynamics in settings where the equilibrium objective is commonly formulated as approximate Nash convergence [Hennes et al., 2020]. In contrast, Φ-AC targets general-sum MARL, where swap-regret-inspired learning naturally relates to correlated-equilibrium conditions and where the main challenge is selecting efficient CE rather than computing a zero-sum Nash solution.

Positioning of Φ-AC. Φ-AC bridges these lines of work by combining scalable centralized-critic MARL with swap-regretbased equilibrium selection. Rather than only improving return maximization or Nash convergence, Φ-AC uses learned swap-regret signals to steer policies toward Pareto-efficient CE while remaining applicable to high-dimensional mixed-motive environments.

3 Preliminaries

We formulate the environment as a general-sumMarkov game and study equilibrium selection through the lens of CE.

3.1 General-SumMarkov Games

We model the multi-agent environment as a GeneralSum Markov Game (MG), formally defined by the tuple G = ⟨N, S, {Ai}i∈N,P, {ri}i∈N, γ,Ω,O⟩. Here, N = {1, . . . ,N} is the set of agents, S is the global state space, P : S × A → ∆(S) denotes the state transition function, ri : S × A → R is the reward function for agent i, and γ ∈ [0, 1) is the discount factor. We assume a partially observable setting where each agent i receives a local observation oi ∈ Ωi through the observation function O : S ×N → Ωi. The joint action is denoted by a = (ai, a-i) ∈ A ≡ ∏i Ai.

3.2 Correlated Equilibrium and OSDP

While NE assumes independent policies, CE [Aumann, 1974] allows coordination via a correlation device, often enabling higher-welfare outcomes. In Markov games, however, equilibrium conditions involve deviations over sequential decision processes rather than a single normal-form interaction. The One-Shot Deviation Principle (OSDP) states that, under standard dynamic-game conditions, it is sufficient to check whether any agent can profit by deviating at a single decision point while following the original policy thereafter [Hendon et al., 1996]. We use this principle as a local approximation of sequential deviation incentives. In our setting, this amounts to measuring whether an agent can improve its return by unilaterally changing its current action, using the stationary Q-function Qπi (s, a) as a local estimate of deviation incentives. We utilize the framework of Φ-Equilibria [Greenwald and Jafari, 2003]. Let Φi be a set of deviation functions ϕ : Ai → Ai for agent i, where each deviation maps an action ai to an alternative action ϕ(ai).

Definition 1 (Φ-Equilibrium [Greenwald and Jafari, 2003]). A joint policy π constitutes a Φ-Equilibrium at state s if no agent i can improve their utility by applying any deviation ϕ ∈ Φi:

Here, (ϕ ◦i π) denotes the joint distribution where agent i follows the deviation ϕ while others follow π. When Φi includes all possible swap deviations, Eq. (1) characterizes CE.

3.3 Convergence via No-Swap-Regret Dynamics

A fundamental result in AGT links regret minimization to equilibrium convergence [Blum and Mansour, 2007; CesaBianchi and Lugosi, 2006]. We define the instantaneous swap regret vector following standard swap-regret formulations [Hart and Mas-Colell, 2000; Blum and Mansour, 2007]. RΦi (s, a) ∈ R|Ai| where each element corresponds to the counterfactual gain of swapping the chosen action ai to a specific alternative a′i ∈ Ai:

where [·]+ denotes the ReLU operator. This vector formulation allows the critic to estimate regret for all possible deviations simultaneously, which is crucial for scalable optimization.

Proposition 1 (No-Φ-Regret to Approximate Φ-Equilibrium [Hart and Mas-Colell, 2000; Greenwald and Jafari, 2003; Blum and Mansour, 2007]). Under standard finite repeatedgame dynamics, let xt ∈ ∆(A) be the joint action distribution induced at round t, and let ui(xt) denote agent i's expected utility under xt. Define RΦ,i T = maxϕ∈Φi ∑Tt=1 [ui(ϕ ◦i xt) -ui(xt)] . If every agent satisfies RΦ,i T /T ≤ ϵ, then the empirical distribution ¯xT = 1 T ∑Tt=1 xt is an ϵ-Φ-equilibrium. When Φi contains all swap deviations, ¯xT is an ϵ-CE.

The Selection Gap. While Proposition 1 establishes that minimizing swap regret leads to approximate CE behavior, it does not determine which equilibrium is selected. In generalsum games, some equilibria may still be socially inefficient or unfair, necessitating an explicit selection mechanism. In Section 4, we address this challenge via a constrained Markov game formulation using a Lagrangian objective that favors socially efficient equilibria.

4 Methodology: Φ-Actor-Critic

We propose Φ-AC, a deep MARL framework designed to encourage efficient coordination through swap-regret-based learning. The framework consists of three components: scalable regret estimation (Sec. 4.1), constrained equilibrium selection (Sec. 4.2), and exploration dynamics (Sec. 4.3). The overall centralized training optimization orchestrating these components is summarized in Algorithm 1.

4.1 Scalability: The Regret-Conditioned Critic

From Simulation to Prediction. The theoretical convergence to CE relies on calculating Instantaneous Swap Regret (Eq. 2). In tabular settings with a simulator, one can simply reset the environment to query "what if" for every unchosen action (O(N|A|) complexity). However, in real-time deployment where resetting is impossible, explicitly computing these counterfactual rewards becomes prohibitively expensive.

Φ-AC solves this bottleneck by training a centralized attention mechanism inspired by MAAC [Iqbal and Sha, 2019] to explicitly predict these counterfactual values. This provides dense regret estimates through a single forward pass, improving scalability in high-dimensional environments. As depicted in Figure 1, we design a dual-head critic to jointly model value estimation and swap-regret prediction. To account for nonstationary learning dynamics induced by concurrently adapting agents, we condition the critic on running cumulative regret Rcum via a Feature-wise Linear Modulation (FiLM) layer, which dynamically scales and shifts critic features [Perez et al., 2018].

The critic consists of two heads:

• Q-Head: Estimates action-values Qψi (s, a,R cum).
• Regret-Head: Predicts the regret vector ˆRψi ∈ R|Ai| , where each element approximates the counterfactual gain of switching from ai to a′i .

To stabilize regret estimation, we compute the regression target

Algorithm 1 Centralized Training Optimization in Φ-AC

• 1: Input: Minibatch B = (o, a, r, o′ ) fromD, Current Cumulative Regret R cum .
• 2: Hyperparameters: δregret (Tolerance), δ (EMADecay rate), ηα (Dual LR).
• // 1. Regret-Conditioned Critic Update (Sec. 4.1)
• 3: Calculate TD targets yi and regret targets R target i using target networks ψ′ , θ′ .
• 4: Get outputs from dual-head Critic conditioned on R cum via FiLM:
• 5: {Qψi , ˆRψi } ←Critic(o, a,R cum )
• 6: Update Critic ψ by minimizing the joint loss:
• 7: LCritic = 1N ∑i (LQ(Qψi , yi) +LR( ˆRψi ,R target i ))
• // 2. Actor & Dual Update via RB-SWO (Sec. 4.2) Sample differentiable actions ˜a ∼ πθ(o) via GumbelSoftmax using regret-biased logits (Eq. (10))
• 8: Get Critic predictions {Qψi (˜a), ˆRψi (˜a)} evaluated at the sampled actions.
• 9: Calculate Regret MagnitudesMi ← ∥[ ˆRψi (˜a)]+∥2.
• 10: [Primal] Update Actor θ by minimizing:
• 11: Ltotal ←LRB-SWO(θ,α) (Eq. (8))
• 12: [Dual] Update multipliers αfair i , αent i via gradient ascent:
• 13: αfair i ← [αfair i +ηα(Mi - δregret)]+
• 14: αent i ← [αent i +ηα(Htarget i -H(πθi ))]+ // 3. Cumulative Regret & Target Networks Update (Sec. 4.3)
• 15: Update Cumulative Regret using Exponential Moving Average (EMA):
• 16: R cum i ← δR cum i + (1- δ) ˆRψi (a)
• 17: Soft update target networks ψ′ and θ′ .

R target i using a Target Network ψ′ with parameters lagging ψ:

The critic minimizes the joint loss, including

4.2 Selection: Lagrangian Dual Optimization

Minimizing swap regret encourages stability, but it does not determine which equilibrium is selected when multiple lowregret equilibria exist. To address this selection gap, we introduce the Regret-Balancing Social Welfare Objective (RBSWO), which maximizes collective welfare while constraining each agent's regret magnitude. This formulation favors highwelfare coordination without allowing the policy to move far from the low-regret region.

Formulation as Constrained Optimization. We formalize RB-SWO as a constrained optimization problem over expected discounted returns. We explicitly define the Regret Magnitude Mi(π) as the expected L2-norm of the positive swap regret vector:

We also define the policy entropy as H(πi) = Eai∼πi [-logπi(ai|oi)], with Htarget i (t) denoting a timevarying minimum entropy target.

The optimization problem is:

where δregret controls the allowed regret magnitude, and Htarget i (t) is annealed over training. We employ the GumbelSoftmax reparameterization trick [Jang et al., 2016] to enable differentiable optimization of the regret constraint.

Lagrangian Dual Optimization. We solve this via the primal-dual method. Introducing the dual variables α = {αfair i , αent i }Ni=1 , where αfair i ≥ 0 controls regret-based regularization and αent i ≥ 0 encourages exploration, we minimize the following objective for actor parameters θ:

The optimization proceeds as an alternating game:

• Primal Update (Actor): Update θ to minimize Ltotal.
• Dual Update (Multipliers): Update multipliers via gradient ascent with step size ηα > 0:

When regret increases, the corresponding dual variable also increases, placing greater emphasis on stability during optimization.

4.3 Dynamics: Entropy Annealing &Warm-up

A practical challenge in regret-based learning is that agents cannot regret actions they have never explored. To resolve this, we implement an explicit schedule:

Discovery (Warm-up). For the first Twarm steps (a predefined hyperparameter), we suppress the fairness penalty (αfair → 0) and enforce a high entropy target. This forces agents to explore the state space purely via Maximum Entropy RL (MaxEnt RL) [Haarnoja et al., 2018], accumulating the reward experiences necessary to define well-defined regret.

Convergence (Annealing). Post warm-up, we activate the regret constraints and linearly decay the entropy, encouraging diverse exploration before convergence to stable coordinated behaviors.

5 Theoretical Analysis

In this section, we provide a theoretical interpretation of Φ-AC by connecting its policy update to no-swap-regret learning and by explaining how the proposed Lagrangian objective biases equilibrium selection. Rather than relying on exact tabular regret computation, Φ-AC uses a learned critic to approximate regret signals in a differentiable deep MARL setting.

5.1 Approximating Regret Matching via Deep Learning

We next interpret the actor update in Φ-AC through the lens of regret matching.

Actor as Smooth Regret Matching. Classically, convergence to CE can be achieved via Regret Matching (RM) [Hart and Mas-Colell, 2000], where agents sample actions proportionally to their positive regret values, i.e., π(ai) ∝ [RΦi (ai)]+. However, the non-differentiability of the standard RM operator makes it unsuitable for gradient-based optimization in deep networks.

To bridge this gap, Φ-AC operationalizes RM via a RegretBiased Softmax Policy. By injecting the predicted cumulative regret ˆR cum into the logits, the policy becomes:

Here, Li(oi, ai) denotes the raw logits output by the actor network prior to softmax normalization, τ is the temperature, and β controls the strength of the regret bias. Lower temperatures make the policy more sensitive to positive regret, while larger β places greater weight on actions with accumulated regret. Together, these terms yield a smooth and differentiable approximation to regret-driven action selection.

This formulation can be interpreted as a smooth approximation of RM. While standard RM selects actions in proportion to positive regret, our softmax-based formulation is conceptually related to exponential weights and Hedge algorithms [Freund and Schapire, 1997; Cesa-Bianchi and Lugosi, 2006], which are also known to induce no-regret behavior under standard online learning assumptions.

Approximate Regret under Function Approximation. In deep MARL, regret estimates are inherently approximate due to function approximation. Following standard analyses of perturbed no-regret dynamics [Cesa-Bianchi and Lugosi, 2006], we consider the setting where the centralized critic provides uniformly bounded regret estimation error: maxs,a ∥Rtrue - ˆRψ ∥∞ ≤ ϵc. Under policy updates that are sufficiently smooth with respect to the regret estimates, the induced perturbation scales with ϵc, implying that the resulting average swap regret remains bounded up to an O(ϵc) approximation term. Consequently, the learned dynamics may approach an approximate CE despite imperfect regret estimation.

5.2 Geometry of Equilibrium Selection

While regret minimization encourages behavior consistent with approximate CE conditions, it does not by itself distinguish high-welfare equilibria from inefficient low-regret outcomes. RB-SWO addresses this selection gap by adding welfare maximization under individual regret constraints, yielding a Lagrangian objective that favors high-welfare coordination.

Dual Forces in Optimization. We formulate equilibrium selection as maximizing social welfare (W = ∑ Qi) subject to stability constraints (Mi ≤ δregret). The Lagrangian objective L(θ, α) = W - ∑ αi(Mi - δregret) induces two distinct gradient forces:

Efficiency Force (∇θW): This term drives the joint policy along the "flat" directions of the CE polytope (where regret is zero) toward higher-welfare regions of the CE set.

Restoring Force (-αi∇θMi): If an agent deviates from equilibrium (increasingMi), the dual variable αi increases (Dual optimization). This increases the optimization pressure toward lower-regret regions.

Fairness via Individual Constraints. Individual regret constraints also play an important role in equilibrium selection. Geometrically, asymmetric equilibria (where one agent yields and another benefits) imply that while the sum of regrets might be low, the individual regret for the yielding agent is high. By enforcingMi ≤ δ for each agent rather than only controlling the average regret, RB-SWO discourages solutions in which one agent absorbs most of the deviation incentive. This encourages more balanced outcomes and reduces winner-takes-all behavior.

6 Experimental Evaluation

While the previous sections introduced the regret-aware architecture and optimization framework of Φ-AC, this section empirically evaluates its ability to learn socially efficient coordination. We evaluate Φ-AC across a hierarchy of complexity: diagnostic matrix games, scalable MPE domains, and Melting Pot Harvest. These settings respectively test equilibrium selection in transparent games, scalability in higher-dimensional interactions, and sustainable coordination under shared resource depletion.

We compare Φ-AC against representative MARL baselines that capture different algorithmic design choices. MADDPG represents deterministic policy-gradient learning, MAPPO and COMA represent stochastic policy-gradient methods with centralized training, and QMIX represents value decomposition under monotonic factorization.

Experimental Setup. Training configurations were tailored to the complexity of each domain: Iterated Matrix Games (IMG) were trained for 1k episodes with a horizon of 25 steps, MPE environments [Lowe et al., 2017] for 30k episodes (horizon 25), and Melting Pot (Harvest) [Leibo et al., 2021] for 10k episodes (horizon 500). To ensure statistical robustness, all results for MPE and Melting Pot are reported as mean ± standard deviations over 5 independent runs.

6.1 Diagnostic Analysis: IteratedMatrix Games

We first employ Iterated Matrix Games to empirically verify the equilibrium convergence of each model. We consider a two-player setting where agents interact repeatedly, allowing them to adapt strategies based on history. We select three canonical games, each representing a distinct coordination failure mode. Fig. 2 visualizes the learning trajectories.

Prisoner's Dilemma. This game tests whether agents can overcome the temptation to defect and coordinate on mutual cooperation. As shown in Fig. 2 (Top-L), MADDPG and MAPPO tend to converge toward the inefficient NE (D,D), while QMIX and Φ-AC reach the social optimum (C,C). Unlike QMIX, which benefits from the monotonic payoff structure in this setting, Φ-AC reaches mutual cooperation through regret-aware coordination.

Chicken. This game tests coordination under two asymmetric equilibria, where either agent can benefit if the other yields. As shown in Fig. 2 (Top-R), baselines often drift toward onesided or unstable outcomes, whereas Φ-AC promotes balanced coordination by reducing regret disparities between agents.

Stag Hunt. This game tests whether agents can coordinate on a high-payoff action that is beneficial only when both agents choose it, rather than falling back to the safer but lower-reward option. As shown in Fig. 2 (Bottom), baselines tend to favor the conservative risk-dominant outcome, while Φ-AC consistently reaches the payoff-dominant cooperative equilibrium.

6.2 Selection under Scale: MPE

We extend our evaluation to the MPE to verify whether the selection capability scales to high-dimensional state-action spaces via our centralized attention critic. We report Mean Reward (efficiency), Cumulative Regret (Convergence), and Regret Gap (|Rcum i -Rcum j |, fairness). Table 1 summarizes the quantitative results, and Figure 3 visualizes the training dynamics.

Cooperative: Avoiding Inefficient Equilibria. In simple spread, agents must cover landmarks while avoiding collisions. COMA and QMIX achieve low regret but converge to lower-welfare solutions, suggesting that stability alone does not imply efficient equilibrium selection. In contrast, Φ-AC achieves the highest reward (-42.24). By balancing welfare maximization with regret constraints, it avoids premature stabilization and reaches a higher-welfare configuration.

Zero-Sum: Stability vs. Exploitation. High individual rewards in simple adversary often signal exploitation rather than equilibrium. MAPPO achieves a high reward (91.17) but suffers from large regret gap (5926.82), indicating a non-stationary cycle where one agent continually exploits a weak opponent without convergence. Φ-AC converges to a reward near zero (-3.27) with significantly lower regret, suggesting a more stable low-regret regime in which unilateral exploitation incentives are reduced.

Mixed-Motive: Scalable Fairness. simple tag involves conflicting goals within a general-sum framework. Baselines exhibit large regret gaps (> 2000), indicating highly imbalanced deviation incentives consistent with winner-takes-all behavior. Φ-AC reduces this gap by nearly two orders of magnitude (53.79). This suggests that the RB-SWOmechanism can scale to high-dimensional domains, enforcing equity by ensuring that the dissatisfaction (regret) is distributed equitably among agents.

6.3 Ablation on Learning Objectives

We dissect the contribution of our proposed learning objectives in simple tag. The results, summarized in Table 2, reveal that each component is critical for preventing unstable learning outcomes.

1. w/o Regret Matching (RM) Bonus: Removing the regret-bias term (β = 0) leads to unstable learning dynamics. The total regret increases substantially to 1933.04 with an extreme standard deviation (± 2007.34). This indicates that without the RM bonus, the learning dynamics fail to converge, oscillating severely or diverging completely.
2. w/o Fairness Objective (RB-SWO): Removing the fairness constraint (αfair → 0) degrades both stability (481.56 → 510.11 in Regret) and efficiency. This suggests that the fairness objective acts as a dual-purpose reg-

ularizer: it not only prevents inefficient outcomes ("Coordination Collapse") but also accelerates convergence by constraining the search space to the stable region.

6.4 Robustness in Dilemmas: Melting Pot

Finally, we evaluate Φ-AC on Harvest, an SSD that serves as a challenging benchmark for sustainable coordination under shared resource constraints. In this environment, apples regenerate based on the density of nearby unharvested apples, creating a "Tragedy of the Commons" where individually greedy behavior can undermine long-term collective productivity. We compare Φ-AC against MAPPO (centralized policy-gradient baseline) and QMIX (value decomposition baseline) using the Sustainability Index (SI), defined as SI = (1 - Gini) × ∑i Returni, which captures the trade-off between productivity and fairness.

Escaping Resource Collapse (Fig. 4 (a,b)). As shown in Table 3, Φ-AC achieves the highest collective reward (134.33) while maintaining competitive fairness relative to MAPPO. In contrast, QMIX suffers from severe inequality and substantially lower collective productivity, resulting in the weakest Sustainability Index. Although MAPPO achieves the lowest Gini coefficient, its lower collective reward leads to a smaller overall sustainability score than Φ-AC. These results suggest that regret-based equilibrium selection helps maintain productive coordination without collapsing into resource-depleting behavior.

Reward-Regret Dynamics (Fig. 4 (c,d)). To further analyze the learned coordination dynamics, we additionally measure the empirical CE gap, defined as the maximum positive swap regret estimated by the critic during evaluation. As shown in Fig. 4(c), the empirical CE gap of Φ-AC remains consistently low throughout training, suggesting reduced unilateral deviation incentives among agents.

This interpretation is further supported by the reward-regret dynamics in Fig. 4(d). During early training, swap regret temporarily increases as agents explore alternative coordination strategies. As learning progresses, however, the regret signal decreases while collective reward remains high. This indicates that agents gradually settle into a stable high-welfare coordination pattern rather than maximizing reward through unstable exploitative behavior.

7 Conclusion

In this work, we introduced Φ-AC, a principled framework designed to integrate game-theoretic swap regret minimization with deep MARL. By shifting the paradigm from reward-only optimization to equilibrium selection, Φ-AC addresses the limitations of conventional MARL objectives in distinguishing efficient coordination from suboptimal stability.

Our empirical evaluation supports Φ-AC across a hierarchy of complexity. At the diagnostic level (Matrix Games), we visualized robust steering toward social optima. At the scalable level (MPE), we demonstrated effective fairness-aware coordination. Most significantly, in the complex SSD (Harvest), we provided empirical evidence of high-welfare coordination behavior consistent with correlated-equilibrium principles.

The observed decrease in smoothed swap regret alongside increasing collective reward suggests that Φ-AC promotes sustained coordination rather than short-term reward maximization. A current limitation is its reliance on discrete action spaces. Future work will extend regret estimation to continuous control, broadening game-theoretic equilibrium selection to physical decision-making domains.