Constructive Conflict-Driven Multi-Agent Reinforcement Learning for Strategic Diversity

1 Introduction

Due to the advancement of deep multi-agent reinforcement learning, many real-world problems have been modeled and addressed as cooperative multi-agent problems [Samvelyan et al., 2019; Sunehag et al., 2017; Rashid et al., 2020], such as traffic signal control [Wiering, 2000], autonomous driving [Hu et al., 2019], and robot control [Haarnoja et al.,

∗Corresponding authors

2019]. The goal of these cooperative multi-agent problems is to maximize the reward from a team perspective [Colby et al., 2015; Rashid et al., 2020]. However, learning effective strategies for such complex multi-agent systems remains a significant challenge. One key problem is that relying on a single feedback signal often leads to homogeneous agent policies, resulting in inefficient exploration and hindering the emergence of complex cooperative behaviors [Li et al., 2021].

Providing additional reward signals [Du et al., 2019; Li et al., 2021; Jiang and Lu, 2021] to individual agents has proven to be an effective approach for diversifying cooperative strategies and improving performance. Existing methods for designing such rewards primarily rely on the mutual information between agent policies and agent identities (IDs) as intrinsic rewards [Li et al., 2021; Jiang and Lu, 2021], but such methods do not consider the influence of other agents and lack intuitive interpretability. LIIR [Du et al., 2019] introduces learnable intrinsic rewards with shared parameters to facilitate information exchange, enabling agents to develop independent strategies. However, the process of learning intrinsic rewards overlooks the important role of competitive mechanisms in cooperative scenarios, as evidenced in sociological research.

There is a unique phenomenon observed in sociological research called Constructive Conflict [Kirchmeyer and Cohen, 1992; King et al., 2009]. Constructive Conflict describes a situation in which the collective intelligence of a cooperative group is stimulated by the positive competition or clash of views between individuals and subgroups. Unlike destructive conflict that hinder the achievement of optimization goals, constructive conflict does not lead to strained relationships or reduced efficiency. Instead, it fosters diverse perspectives and strategy refinement, positively contributing to group goals. Inspired by the concept of constructive conflict, we propose competitive intrinsic rewards to enhance agent learning and team performance. In particular, unlike intrinsic rewards, the competitive intrinsic rewards provide cooperative agents with competitive incentives, thereby stimulating strategic communication and strategy diversification among agents.

In this paper, we propose an algorithm called CoDiCon, which is designed for competitive intrinsic rewards. Specifically, ranking is an effective method to encourage mutual competition, so we design an intrinsic reward that incorporates ranking property. The overall algorithm adopts an actorcritic structure [Lowe et al., 2017; Schulman et al., 2017] based on MAPPO [Yu et al., 2022], where each agent has a separate set of parameters. The global critic evaluates the current action values, and a centralized intrinsic reward generation module produces a ranked intrinsic reward for each agent's current action to foster competition. The agent's policy is jointly optimized using both extrinsic and intrinsic rewards, with the intrinsic reward serving merely as a training signal to distinguish rankings rather than having any inherent meaning. Mathematically, the optimization problem can be modeled as a constrained bilevel optimization problem, where the constraints in the outer optimization ensure the equivalence between the optimization objective and the original maximization of the environmental reward. We summarize our contributions as follows:

• We design an effective intrinsic reward mechanism based on the principles of constructive conflict, introducing competitive incentives to cooperative multi-agent systems to enhance team performance.
• The ranking module is proposed to provide agents with intrinsic rewards possessing ranking property. The optimization objective is formulated as a constrained bilevel problem, ensuring that optimizing the policy based on the ranking rewards aligns with the original goal of maximizing environmental rewards.
• Experimental results demonstrate that our algorithmoutperforms existing methods. Visualizations of the agents' intrinsic rewards and state-reward space indicate that the learned intrinsic rewards produce distinct signals, enabling agents to take diverse actions collaboratively.

2 RelatedWork

In recent years, deep multi-agent reinforcement learning has made significant advancements. Research efforts such as COMA [Foerster et al., 2018], MADDPG [Lowe et al., 2017], and LICA [Zhou et al., 2020] have explored policy-based approaches to multi-agent problems, utilizing a centralized critic to evaluate the value of each distributed policy. Value decomposition methods, including VDN [Sunehag et al., 2017], QMIX [Rashid et al., 2020], and QTRAN [Hostallero et al., 2019], decompose environmental feedback into individual agent value functions, thereby enabling credit assignment. QPLEX [Wang et al., 2020] proposed using dueling networks to relax the monotonicity constraint of mixing networks, expanding their representational capacity. Learning from others is an innate human survival skill, a phenomenon mirrored in agent policies, where information exchange via mixing networks enhances policy generation. However, most existing methods focus on centralized mixing network structures to satisfy value function constraints, often relying on strong assumptions about these functions. In contrast, our approach learns explicit intrinsic rewards for each agent at each time step, avoiding such assumptions and enabling immediate credit assignment.

Our work addresses the problem of designing intrinsic rewards in cooperative multi-agent systems, a topic that has been explored in some previous studies. EOI [Jiang and Lu, 2021] proposed learning a classifier for observations to compute the probability that an observation belongs to each agent, using this probability as an intrinsic reward to adjust the agent's final reward. CDS [Li et al., 2021] introduced maximizing the mutual information between agent IDs and trajectories as an intrinsic reward to promote diverse policies. However, such methods do not consider the influence of other agents, overlooking the importance of learning from them, and they lack intuitive interpretability. LIIR [Du et al., 2019] designed a module for generating intrinsic rewards to guide more diverse policies, ensuring that the two-level optimization is equivalent to the initial optimization problem. However, it also overlooks the positive competitive influence that other agents can have on policy development. Furthermore, directly calculating intrinsic rewards can result in an excessively large optimization space, increasing the risk of policies converging to local optima.

3 Preliminary

3.1 Cooperative Multi-Agent Reinforcement Learning

Multi-agent reinforcement learning [Sutton and Barto, 2018] extends traditional reinforcement learning to address decision-making problems with multiple agents in sequential environments. A fully cooperative multi-agent problem can be represented as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP) [Oliehoek and Amato, 2016]. In this model, a Dec-POMDP is described by the tuple ⟨S, A, U, Z, O, P, r, n, γ⟩. Here, S represents the global states of the environment, and A denotes the set of n agents. At each time step t, each agent i ∈ A selects an action ui from its action set Ui, resulting in a joint action u ∈ U ≡ U n . The state transition function P determines the next state s′ ∈ S based on the current state s and the joint action u. All agents share a common reward r(s, u), which depends on the state and joint action. A discount factor γ ∈ [0, 1) is used to weigh future rewards. In the partially observable setting, each agent receives a local observation zi ∈ Z, derived from the observation function O(s, u) : S × U → Z, where Z is the observation space. Each agent learns an individual policy πi(ui|τi; θi) with parameters θi based on its own actionobservation history τi. The agents' combined policies determine the joint action taken in the environment.

3.2 Centralized Training with Decentralized Execution

Centralized Training and Decentralized Execution (CTDE) is a commonly used paradigm for addressing multi-agent problems. In this paradigm, actor-critic methods are often chosen [Williams, 1992; Andrychowicz et al., 2016; Schulman, 2015; Sutton and Barto, 2018; Yu et al., 2022]. In our approach, we choose the MAPPO [Yu et al., 2022] algorithm as the base framework, which consists of n independently parameterized policies πθi . The policy parameters θ = {θ1, θ2, ..., θn} are optimized using policy gradient methods to maximize the extrinsic reward:

where p(θ) = πθ(u|s) πold,θ(u|s) denotes the policy ratio, which depends on the policy parameters θ, and ˜ p(θ) = clip(p(θ), 1 - ϵ, 1 + ϵ) is the clipped policy ratio. Aπ(s, u) represents the advantage function based on the state and actions. There are several methods to estimate Aπ(s, u). For example, Aπ(s, u) = rex (s, u) + V ex (s′ ) - V ex (s) is the standard advantage function [Schulman, 2015], where s′ is the next state.

4 Method

In this section, we present the competitive intrinsic reward algorithm, CoDiCon. First, we provide a formal definition of the problem and represent it as a constrained bilevel optimization problem. Then, we detail the approach for solving this problem, which effectively addresses the interplay between agents' learning and their competitive interactions.

4.1 The Optimization Objective

We define the components of the agent's reward, which consist of intrinsic reward rin and extrinsic reward rex . The intrinsic reward is parameterized by η. At each time step, each agent takes a state-action pair as input, the hybird reward can be expressed as:

In equation (2), λ represents the hyperparameter that balances intrinsic and extrinsic rewards. It is important to note that the additional intrinsic reward does not appear in the standard multi-agent problem. After defining a hybird reward rhybird i,t for each agent at each time step, we define the discounted hybird reward for each agent as follows:

and the hybird value function for agent i is defined as:

Unlike the extrinsic value function V ex , these hybird value functions V hybird i do not have any actual physical meaning. They are just used to update the parameters θi of each agent's policy. Next, we consider the overall optimization objective, defined as:

where J hybird i = Es0,ui,0,... [Rhybird i,0 ] depending on θi and η, η indicates the parameter of the ranking module and θ indicates the policy parameter set {θ1, θ2, · · · , θn}. The updates of the ranking module parameters and policy parameters are performed alternately. The ranking network module consists of fully connected layers and a sorting function layer, ensuring that the intrinsic rewards rin it outputs have numerical differences and are arranged in ascending order. When the parameters of the ranking module are frozen, the policy parameter θi is optimized by maximizing the hybrid expected cumulative return J hybird i for agent i. The advantage of this approach lies in using the ranked intrinsic rewards at each step for policy learning, where competition is captured by the intrinsic rewards, defined through pairwise performance ranking as rin i = rankη(i,-i). This mechanism motivates agents to compete and learn from each other, leading to more complex and diverse behaviors. Conflict arises when an agent's action that maximizes its intrinsic reward rin i opposes the team's extrinsic reward rex . Specifically, conflict exists if for some agent i, ∇ui rex · ∇ui rin i < 0, implying the agent's optimal behavior hinders team performance. From an optimization perspective, problem (5) can be viewed as a constrained bilevel optimization problem, where the outer optimization constrains the inner policy improvement process. The ultimate goal of optimization is to maximize the environmental reward under the constraint of intrinsic rewards. This ensures alignment with the original objective of maximizing the environmental reward, allowing better-performing policies to receive higher intrinsic rewards. In the next section, we will discuss the relationship between J ex and the intrinsic reward parameter η during the optimization process.

4.2 Algorithm

For the constrained two-level optimization problem, at each update step, the policy parameters are updated based on the hybrid expected return, while the update of the ranking module parameters depends on the ranking process and the extrinsic expected return.

Specifically, the parameters of each agent are updated using the hybird critic network. Given the trajectory data generated by the agents, the policy parameters are updated through the policy gradient method as described in (1):

where p(θi) is the policy ratio of agent i. Ahybird i (s, u) denotes the hybird critic, which can be chosen in various ways [Sutton and Barto, 2018; Schulman, 2015; Yu et al., 2022]. In this paper, we choose Ahybird i (s, u) = rhybird i (s, u)+ V hybird φ (s′ ) - V hybird φ (s) as the advantage function, where V hybird φ represents the hybird state value, parameterized by φ, and s′ denotes the next state of the agent in the trajectory. Given (6) and a policy learning rate a, the updated policy parameter θ′i can be expressed as: θ′i = θi + α∇θi min (p(θi)Ahybird i (s, u) , ˜ p(θi)Ahybird i (s, u)).

To ensure that intrinsic rewards promote competition among agents, we update η by applying a relative numerical constraint to the intrinsic rewards produced by the ranking module. This ensures that the rewards follow a sequential distribution. Specifically, we use mean squared error (MSE) loss and variance loss of intrinsic rewards to maintain the output as a sequence with numerical differences:

where y is the optimization target for the intrinsic reward, represented as an ordered sequence. This sequence is randomly initialized at the beginning of training (in our setup, 20% positive values and 80% negative values) and remains fixed during subsequent training. ¯ rin is the mean of the intrinsic rewards. For the update of the parameter η, we first minimize the MSE loss and maximize the variance loss:

Given (9) and a learning rate β, the updated parameters of the ranking module can be expressed as: η′ = η - β∇ηL(η).

Next, we construct expressions for η′ and J ex to update the parameter η′ . Using the updated policy parameter θ′ , we apply the chain update rule to get:

Algorithm 1 The algorithm of CoDiCon.

• 1: Input: Policy learning rate α and intrinsic reward learning rate β.
• 2: Initialize: Policy parameters θ and intrinsic reward parameters η.
• 3: for t = 1 to Tmax do
• 4: Sample a trajectory D = {s0,u0, s1,u1, · · · } by executing actions with the decentralized policies {πθ1 , · · · , πθn };
• 5: Update θ according to (6) with learning rate α;
• 6: Update η according to (9) with learning rate β;
• 7: Compute (11) and (12) using samples fromD after the first update of η;
• 8: Update η′ according to (10) and step 7 with learning rate β;
• 9: end for
• 10: return policy parameters θ

The purpose of (10) is to formally express the impact of J ex on the updated policy parameter η′ through the updated parameter θ′i . This technique is widely adopted in meta-gradient learning [Andrychowicz et al., 2016; Santoro et al., 2016; Xu et al., 2018]. Using samples generated by the updated policy network, the meta-gradient ∇η′ can be computed. In (10), ∇θ′i J ex can be estimated by stochastic gradient as

here, Aex (s, u) denotes the centralized extrinsic critic. Similar to the centralized hybrid critic, we define Aex (s, u) = rex (s, u) + V ex ϕ (s′ ) - V ex ϕ (s), where V ex ϕ (s) is the extrinsic value function with parameters ϕ. For the update of η′ , the second term in (10) can be derived as:

Figure 1 presents the overall framework of the CoDiCon algorithm. A detailed description of the algorithm is provided in Algorithm 1.

5 Experiments

In this section, we analyze and illustrate the performance and effectiveness of our algorithms in Pac-Men, Google Reasearch Football (GRF) and Starcraft Multi-agent Changellenge (SMAC) environments. We select the multiagent policy-based method (MAPPO), the intrinsic reward policy-based method (LIIR), and the independent emergence methods (EOI, CDS) as baseline methods for comparison. First, we analyze the effectiveness of introducing a competitive mechanism to improve policy performance in the PacMan environment. Next, we conduct performance comparison experiments on GRF and SMAC. Then, we perform a case study on GRF to analyze the effectiveness of the intrinsic rewards in our algorithm. Additionally, we use t-SNE visualization in the SMAC environment to compare the final distribution of policies in the state-reward space, explaining the intrinsic reasons behind the effectiveness of the algorithm. Finally, we experimentally validate the two modules of the algorithm that encourage intrinsic reward variability through ablation studies.

5.1 Competitive Intrinsic Reward

We design the Pac-Men environment, shown in Figure 2a, to demonstrate how our algorithm encourages agents to compete, thereby improving overall algorithm performance. In this environment, the central room connects four equally sized rooms and is equidistant from each of them. Four agents are initialized at fixed positions near the entrance of each room, and each agent has a 5x5 field of view. To foster competition among agents, the lower room contains the highest number of "dots" compared to the other three rooms. To make the environment more challenging, the game is limited to 17 timesteps, making it difficult for a single agent to collect all dots in the lower room, requiring at least two agents. The best case scenario is that one of the agents gives up the closest room and chooses to enter the room below, and then the two agents eat all the dots in the room below together. Agents incur a -0.25 penalty per timestep but gain 1 reward for each bean eaten. We compare our algorithm with SOTA algorithms. As shown in the visitation heatmap in Figure 2b and 2c, the EOI agents dispersed to four different rooms, whereas two agents in our algorithm both moved to the lower room, which contains the highest number of dots. In this environment, overly dispersed strategies do not result in optimal rewards. The experimental results shown in Figure 2 demonstrate that our algorithm successfully discovers the optimal strategy, while other algorithms fall into local optimal solutions. The possible reason is that our algorithm enables other agents to learn the high-reward strategy of the agent that moves to the lower room to collect "dots" by continuously ranking the intrinsic rewards, that is, by assigning higher intrinsic rewards to the agents that go to the lower room.

5.2 Performence on GRF

In this section, we first evaluate the performance of the algorithm in the GRF environment. Specifically, we compare our algorithm with others on GRF tasks of increasing difficulty, including academy 3 vs 1 with keeper, academycounterattack easy, and academy counterattack hard. In the GRF task, the agents need to collaborate in time and space to organize the attacking opportunities, and only the scoring are rewarded. In our experiments, we control the agents on the left (in yellow) except the goalkeeper's agent, and the agents on the right are controlled via the roule-based AI built into the game engine. The agents have 19 discrete action spaces, including moving from different angles, sliding, shooting, and passing. Observations include the position and direction of movement of the agents and other agents, as well as the position and direction of movement of the ball. The movement of the soccer ball in the z-axis direction is also included in the observations, and the agents receive feedback from the environment only at the end of the game.

Our algorithm outperforms all other algorithms in all test environments, and the results are shown in Figure 3. In the academy 3 vs 1 with keeper, other baselines also perform well, such as MAPPO and LIIR, showing performance close to that of our algorithm. However, on the progressively more difficult environments of academy counterattack easy and academy counterattack hard, the advantage of our algorithm gradually widens, and the interval between the training curves becomes more and more obvious. This demonstrates that our algorithm is able to effectively capture valuable states and actions in sparse reward-difficult environments, generating intrinsic rewards that encourage the agent to learn better.

5.3 Performence on SMAC

In this section, we test our algorithm in the StarCraft Micromanagement (SMAC) environment, a popular MARL benchmark. Each agent has attributes like health, weapon CD, shield, unit type, last action, and relative distances to observed units, with enemy units sharing similar attributes except CD. In partially observable settings, agents gather information within a circular range. The action space includes four movement directions, k attack actions (where k is the max number of enemies), a stop action, and a no-op action, with invalid actions masked. We evaluate on scenarios: 3s vs 5z, 8m vs 9m, MMM2, 2c vs 64zg, 5m vs 6m, 3S5Z. Only the Zealot is melee, the Medivac is a non-attacking support unit, and others are ranged. Agents receive a joint team reward based on total damage, with a significant bonus for winning.

We present the experimental results of the comparison in Figure 4. Our algorithm outperforms the baseline algorithms in most environments, with performance comparable to EOI only in the environment of 2c vs 64zg. Both CDS and EOI achieve performance close to our algorithm on some maps, such as 3s vs 5z and 8m vs 9m. However, their performance on 5m vs 6m and 2c vs 64zg are highly unstable, indicating that the diversity mechanisms in CDS and EOI lack effective exploration and utilization of the state space, which may lead to local optima. MAPPO shows relatively stable and high performance across all environments, but it still lags behind our algorithm in terms of training efficiency and convergence speed. In contrast, LIIR performs poorly on all SMAC maps, suggesting that the unconstrained learnable intrinsic rewards in LIIR make learning highly unstable, particularly in scenarios with denser rewards. In contrast, our algorithm provides stable initial feedback during training by introducing a distinctly different prior intrinsic reward.

5.4 Visualizing the Learned Intrinsic Reward

The effectiveness of the strategy is demonstrated on the GRF counter attack hard mission, where we control four players positioned at different locations, cooperating to kick the ball from the half-field area into the opponent's goal. During the course of the play, the team encounters two opposing players controlled by the built-in AI, as well as a goalkeeper defending the opponent's goal. We visualize the trajectories of the players and the ball in Figure 5 (left), where the players' paths are represented by white arrows and the ball's path by yellow arrows. Agents are distinguished by numerical identifiers. Player 2 dribbles the ball and passes it to player 3, who receives it and shoots it into the goal. Player 1 collaborates with player 2 in the attack, applying pressure on the two opposing defenders, while player 0 moves toward the penalty area to support the offensive play. In Figure 5 (right), we plot the intrinsic reward curves corresponding to each player during the passing and shooting process.

• In the passing process, higher rewards are assigned to player 3 (the player taking the shot), player 2 (the player dribbling the ball), and player 1 (the player supporting the attack). In contrast, player 0, who is farther from the

t-SNE 1

Figure 6: t-SNE results on the state-reward space.

offensive play, receives lower rewards.

• After player 2 passes the ball, his intrinsic reward drops sharply. During the flight of the ball, player 1 maintains a relatively high intrinsic reward because he is close to the two opposing defenders and supports the offense. Once player 3 receives the ball and takes the shot, the intrinsic rewards of the three players involved in the offensive play increase significantly.

This demonstrates that the intrinsic rewards in our algorithm effectively evaluate each agent's state and its contribution to the overall team performance.

5.5 Strategy Visualization in State-Reward Space

In addition to evaluating the performance of the trained policies, we are also interested in understanding how the learned intrinsic reward function influences policy learning. To analyze the intrinsic impact of our algorithm on policy training, we combine the state space and reward space to form a highdimensional state-reward space. In this space, we can intuitively compare the distribution of the learned policy across states and the rewards obtained by executing actions in these states. We use t-SNE [Van der Maaten and Hinton, 2008] to perform a low-dimensional mapping of the state-reward space, where the specific value of each point is represented by the environmental reward. Different colors are used on the graph to represent the values, with warmer colors indicating higher values, as shown in the figure 6. We compared CoDiCon, MAPPO, and the intrinsic reward algorithm without ranking (CoDiCon without rank) based on the results of the trained policies over 50 episodes. As shown in the figure, MAPPO explores more meaningless states and attempts actions over a larger area (larger canvas size) compared to algorithms utilizing intrinsic rewards, but this does not lead to higher rewards. In contrast to the policy without ranked intrinsic rewards, our algorithm avoids meaningless exploration in adjacent feature spaces with high feature similarity (the points are more dispersed yet relatively concentrated)

and identifies high-value actions at key nodes (points with more warm colors). The possible reason is that our algorithm facilitates mutual learning among agents for high-reward actions, and the differences in intrinsic rewards make the strategy more complex, diverse, and efficient.

5.6 Ablation Study

In our algorithm, two loss functions are designed to promote intrinsic reward variability and constrain the training of the ranking module parameters: the priori intrinsic reward MSE loss and the intrinsic reward maximum variance loss. To evaluate the effectiveness of these constraints, we conduct ablation experiments on the two loss functions as shown in Figure 7. Here, w/o pri refers to the algorithm without the priori loss constraint, w/o var refers to the algorithm without the variance loss constraint, and w/o rank refers to the algorithm without both loss constraints. We perform comparative experiments on the 5m vs 6m and MMM2 scenarios in SMAC, and the results in both scenarios are consistent. These results indicate that both loss functions make positive contributions to policy training, outperforming the performance achieved without the intrinsic reward ranking loss, thereby demonstrating that both constraints are effective. Notably, using only the priori intrinsic reward constraint achieves higher performance and faster convergence compared to using only the variance constraint. A possible explanation is that reinforcement learning involves a large number of low-value states during the early stages of training, where distinguishing these low-value states and assigning them lower intrinsic rewards is critical. In contrast, variance constraints primarily become effective during the later stages of training.

6 Conclusion

We propose a novel competitive intrinsic reward multi-agent algorithm, CoDiCon, which enables each agent to learn an intrinsic reward with ranking properties. This approach effectively guides mutual competition and learning among agents, enhancing the efficiency of learning diverse strategies even when the environment provides only a single team reward. Our algorithm is formulated as a constrained bilevel optimization problem, theoretically ensuring that the final optimization objective aligns with maximizing the original environmental rewards. Experimental results on the SMAC, GRF, and Pac-Men environments demonstrate that CoDiCon outperforms existing state-of-the-art methods in terms of performance. Furthermore, case studies further validate the effectiveness of our intrinsic reward design.

Acknowledgments

This work is jointly supported by the National Science and Technology Major Project (No.2022ZD0116403), the Key Research Project of Chinese Academy of Sciences (No.RCJJ145-24-15), the Postdoctoral Fellowship Program of CPSF (Grant No.GZC20232995), the China Postdoctoral Science Foundation (Grant No.2024M763533).