Reexamining the Exploration–Exploitation Dilemma from an Entropy-Driven Perspective

1 Introduction

In reinforcement learning (RL) [Zhao et al., 2022; Gan et al., 2024; Yan et al., 2026], the two fundamental components, i.e., exploration and exploitation, are simultaneously opposing and complementary. Too much exploration reduces learning efficiency, while excessive exploitation of uncertain knowledge hinders long-term reward maximization, leading to suboptimal policies [Tokic, 2010]. How to achieve an optimal balance of exploration and exploitation remains a significant challenge, known as the exploration-exploitation dilemma [Ahmed et al., 2019; Schulman et al., 2015, 2017; Bellemare et al., 2016].

Methods based on policy and rewards have been proposed to address exploration-exploitation trade-offs in RL. For example, ϵ-greedy and its variations [Sutton and Barto, 1999; Tokic, 2010; Mnih et al., 2015] randomly select exploratory and exploitative actions, which is simple to implement and broadly effective [Taiga et al., 2020; Dabney et al., 2021; Dann et al.,

∗Corresponding author

2022]. Intrinsic rewards [Schmidhuber, 2010; Kim et al., 2019; Ermolov and Sebe, 2020; Bougie and Ichise, 2020; Mu et al., 2022; Henaff et al., 2022; Mutti et al., 2022] driven by state uncertainty or novelty measurement incentivize and guide the direction of exploration. Entropy has been empirically used in policy optimization to directly encourage exploration [Ahmed et al., 2019; Haarnoja et al., 2018; Zhang et al., 2024; Han and Sung, 2021]. However, existing methods pay less attention to the balance between exploration and exploitation, either leaning toward one side of the exploration-exploitation spectrum or relying on manually designed heuristics to gradually reduce exploration. None of them has achieved a principled adaptation between exploration and exploitation.

In this paper, we revisit the exploration-exploitation dilemma from an entropy-driven perspective. We start by elucidating the relationship between entropy and exploratory motivation, using entropy as a proxy to understand the tradeoff between exploration and exploitation. We then prove that an extended Bellman equation with a novelty-seeking term guarantees appropriate entropy adjustment and its globally monotonic decline. Hence, we theoretically formulate a dynamic adaptive process of policy optimization that accommodates all three cases of the exploration-exploitation trade-off and ultimately transits from exploration to exploitation.

Based on the theoretical findings, we instantiate an end-toend adaptive framework AdaZero, which adaptively balances the strength of exploration and exploitation. A state autoencoder takes real-time state images as input and outputs the reconstruction error as an exploratory reward to encourage exploration. The reconstructed state image generated by the autoencoder, which represents the agent's mastery level of the current state, is fed into an evaluation network to determine the strength of exploration and exploitation. Guided by this mechanism, agents can automatically adapt to environmental changes and decide how much to explore or exploit.

To provide empirical support for our theoretical findings and validate the effectiveness of our proposed self-adaptive mechanism, we conducted experiments on many Atari and MuJoCo environments, considering both continuous and discrete action state spaces. The results indicate that AdaZero tends to outperform previous methods across environments ranging from simple to complex, highlighting the potential effectiveness of our self-adaptive mechanism. Especially in Montezuma, AdaZero boosts the final returns by up to fifteen times. We also perform a series of visualization analyses to reveal the functionality of entropy in policy optimization and how AdaZero's automatic adaptive exploration and exploitation effectively works. In summary, this paper makes the following three innovations:

• It establishes a theory basis for quantifying explorationexploitation trade-off through the lens of entropy.
• It proves an optimization scheme naturally accommodating trade-off with global convergence of exploration, guiding the design of a principled self-adaptive policy.
• It instantiates the theoretical findings as an adaptive framework AdaZero and validates its strong generalizability through extensive experiments.

2 RelatedWork

2.1 Sparse Rewards Tasks

Sparse reward [Gallou´edec and Dellandr´ea, 2023; Kim et al., 2023; Silvia, 2012; Barto, 2013; Bellemare et al., 2016; Ostrovski et al., 2017; Pathak et al., 2017; Burda et al., 2018; Yan et al., 2024] is a typical challenge in reinforcement learning and is widespread in real-world applications such as robot control, autonomous driving, gaming, and complex task planning. The characteristic of such environments is that the agent has limited opportunities to receive reward signals unless it reaches specific goals or triggers certain events. As a consequence, the agent receives little effective feedback most of the time, leading to a drastic reduction in policy optimization efficiency and making it difficult to determine which actions or state sequences lead to the eventual reward [Mnih et al., 2013; Watkins, 1989]. Therefore, enhancing exploration in sparse-reward environments remains a critical challenge in reinforcement learning.

2.2 Exploration-EnhancingMethods

In sparse-reward environments, mainstream approaches can be broadly divided into two categories: methods based on maximum entropy and methods that use intrinsic rewards, both of which aim to expand action and/or state exploration.

The core idea of entropy-based reinforcement learning [Arriojas et al., 2023; Haarnoja et al., 2018; Ahmed et al., 2019; Zhang et al., 2024; Han and Sung, 2021] is to apply entropy regularization to policy optimization in order to maintain policy diversity. By incorporating an entropy term into the objective function, agents are encouraged to explore a broader range of actions, thereby avoiding local optima. This mechanism enhances the agent's exploration capabilities, making it more adaptable in sparse reward environments.

Another stream of research focuses on manipulating the rewards, particularly the exploratory rewards [Burda et al., 2018; Ladosz et al., 2022; Chen et al., 2022], which are added to the original exploitative rewards from the environment. To balance the effect of the two types of rewards, a decaying scheme is introduced for the exploratory rewards [Stadie et al., 2015; Badia et al., 2020; Guo et al., 2022]. However, these methods fail to achieve an optimal balance between exploration and exploitation, which may lead to an effective learning process [Taiga et al., 2021]. Especially when the environmental rewards are sparse, the ever-existing exploratory rewards lead to collapsed total rewards and aimless exploration.

2.3 Exploration-Exploitation Decoupled RL

Decoupled RL methods [Colas et al., 2018; Sch¨afer et al., 2021; Whitney et al., 2021; Liu et al., 2021], which explicitly separate the exploration and the exploitation processes and train them independently. Through this approach, the agent can focus on specific tasks during different learning phases.

However, despite the theoretical advantage of decoupled RL methods, some challenges remain in practice. Amajor issue is that a rigid separation and periodically switching between exploration and exploitation can lead to drop in efficiency. The agent may fail to fully utilize the information gathered in the exploration phase for exploitation, or waste resources in exploration neglecting actual strategy requirements. Moreover, the coordination mechanism of exploration and exploitation is not well designed. The agent may struggle to adapt flexibly to changes in the actual environment, making the dynamic balance between exploration and exploitation more difficult. Thus, how to effectively coordinate the alternation and collaboration between exploration and exploitation in practical applications remains to be further researched.

2.4 Exploration-Exploitation Trade-off

At the policy level, attempts have been made to incorporate exploration with exploitation, the representative of which is ϵ-greedy and its variations [Dabney et al., 2021; Taiga et al., 2021]. While this method generally applies to a wide range of environments, it mainly relies on manually tuned hyperparameters or a specifically designed prior.

2.5 Atari andMuJoCo Benchmarks

In RL research, Atari [Towers et al., 2026] and MuJoCo [Ouhssain et al., 2026] are two of the most commonly used and representative benchmark environments. Atari games are typically based on high-dimensional pixel observations and involve discrete action spaces, complex state transitions, and varying degrees of reward sparsity. Thus, they are widely used to evaluate agents in visual decision-making, long-horizon exploration, and sparse-reward scenarios. In particular, challenging tasks such as Montezuma's Revenge require long sequences of effective exploration to obtain rewards, making them important platforms for examining exploration mechanisms.

In contrast, MuJoCo focuses on continuous control tasks, where agents are required to learn stable, smooth, and highprecision control policies in physical simulation environments. These tasks involve continuous state and action spaces, with reward signals usually related to motion efficiency, posture stability, energy consumption, or distance to the target. Therefore, MuJoCo is well suited for evaluating continuous-action decision-making, dynamic system modeling, and policy optimization stability. Using Atari and MuJoCo together enables a comprehensive evaluation of RL algorithms across discrete and continuous control, visual decision-making and physical control, as well as sparse and dense reward settings.

3 Preliminaries

Reinforcement Learning. Reinforcement Learning [Yan et al., 2025; Wu et al., 2024; Zhao et al., 2021] is commonly modeled as a Markov Decision Process (MDP) [Sutton and Barto, 1999; Wu et al., 2023], it is described by the tuple ⟨S,A, P,R, γ⟩, where S and A represent state and action spaces respectively. The function P is the transition probability function, indicating the likelihood of moving from one state to another following an action. The function R is the reward function. The discount factor γ ∈ [0, 1) modifies the value of future rewards. The agent's interactions within the MDP generate a trajectory τ of states, actions, and rewards: s0, a0,R(s0, a0), · · · , st, at,R(st, at), · · · . The state-action value function Qπ (st, at) is defined as follows:

Let π(a|s) be the policy determining the probability of taking action a in state s. The policy is defined in the softmax form of the Q value:

The entropy of the policy π is defined as:

To guide the agent to make decisions that maximize long-term gains, the goal of policy optimization is to learn a policy π that maximizes the expected total discounted reward η(π), defined as follows:

Exploration-Exploitation Dilemma. Extending the traditional MDP framework to incorporate exploration, new rewards are introduced to help agents explore sparse reward tasks. The total reward function Rtotal combines exploitative rewards Rexploit (extrinsic rewards corresponding to above R in MDP) and exploratory rewards Rexplore (intrinsic rewards):

The Q-function is updated with exploratory rewards added:

where Qexploit(s, a) = Eτ [∑i γ iRexploit(si, ai)|s, a], and δ(s, a) = λEτ [∑i γ iRexplore(si, ai)|s, a] is the accumulated return with only exploratory rewards for simplicity.

Note that the coefficient λ is empirically determined in existing research due to the inherent challenge of the explorationexploitation trade-off. Moreover, as Rexploit is generally sparse in practice, optimization will be biased toward Rexplore, leading to a non-optimal policy.

4 Exploration-Exploitation Adaptation from the Perspective of Entropy

Probing Exploration-Exploitation Trade-off Through Entropy. We first investigate the relationship between entropy and exploratory behavior. According to Eqn. (2), we can define the policies without and with exploratory motivation, i.e., πexploit and πtotal respectively, which are the softmax over Qexploit and Qtotal. We prove that under mild conditions, entropy and exploratory rewards consistently increase or decrease together during policy optimization. Given an action space A of N actions, suppose a1 ∈ A is the optimal action and a2 ∈ A is the suboptimal action, and so on, i.e. Qexploit(s, a1) ≥ Qexploit(s, a2) ≥ · · · ≥ Qexploit(s, aN). For the convenience of subsequent proofs, we define the following abbreviations for the commonly used terms:

where Qi = Qexploit(s, ai) and δi = δ(s, ai).

Lemma 4.1. For ∀j > i, if 0 ≤ δ(s, aj) - δ(s, ai) ≤ Qexploit(s, ai) -Qexploit(s, aj) holds, for any s, we have

Lemma 4.2. (Connection between entropy and exploration.) Under the condition of Lemma 4.1, for any s, we have

The proofs of Lemma 4.1 and 4.2 are shown in Appendix. Lemma 4.2 demonstrates that incorporating exploratory rewards leads to increased entropy, thereby enhancing the agent's exploration capability. Hence, we gain a theoretical understanding of the exploration-exploitation trade-off through the lens of entropy.

The condition of Lemma 4.1 indicates that when δ(aj) - δ(ai) > 0, the probability of non-optimal actions increases, thereby encouraging the exploration of more non-optimal actions. However, the exploration should not be aimless. Thus, we assume its upper bound to be Qexploit(ai) -Qexploit(aj). The above condition is mild and has already been satisfied in existing classic methods like discrete SAC [Haarnoja et al., 2018] and RND [Burda et al., 2018].

Self-Adaptive Bellman Equation. As depicted in Eqn. 3, in current RL methods with a fixed coefficient, the exploration continues to play a role even when it is no longer necessary. Hence, we formulate a new adaptive mechanism for dynamically balancing exploration and exploitation. We theoretically prove how its exploration can be adjusted towards both sides of the exploration-exploitation spectrum in a principled manner.

Specifically, we extend the original Bellman equation with a novelty-seeking term:

where Qexploit(s, a) is the same as in Eqn. (3), ˆ δ(s, a) = Eτ [∑i γ i (1-α(si))Rexplore(si, ai)|s, a] represents the strength of exploration, and α(si) ∈ [0, 1] is a function to balance the trade-off between exploration and exploitation on each state si.

Based on Eqn. (4), we can derive a policy optimization process that inherently accommodates all three regimes of the exploration-exploitation spectrum from the perspective of entropy. In contrast to our method, as shown in Lemma 4.2, existing methods can only maximize entropy to foster exploration without the capability to adaptively increase or decrease the strength of exploration.

Theorem 4.1. Under conditions of Lemma 4.1 and different values of α function, for any s, policy optimization with the above Eqn. 4 can have either

Proof. According to Eqn. (4), we have

Depending on the mastery level of the states α(s), there will be different scenarios regarding the relationship between exploration and exploitation. We mainly discuss the following three typical cases:

Case I - Exploration Dominant: for any s, when α(s) ≡ 0, Qtotal(s, a1) = Qexploit(s, a1) + ˆ δ(s, a1), and Qtotal(s, a2) = Qexploit(s, a2) + ˆ δ(s, a2). From lemma 4.1, we know H(πexploit|s) < H(πtotal|s), i.e., the entropy increases. In this scenario, the strength of exploration reaches the peak level, which means our method exhibits stronger exploratory capabilities than methods without intrinsic rewards.

Case II - Adaptive Exploration-Exploitation: According to Eqn. (5), let α(s) = 1, we have

Here we discuss two cases. For the first case, when the subsequent transition state space corresponding to action a1 differs from that of action a2, the following holds:

and

For the second case, If the subsequent transition state spaces corresponding to a1 and a2 are the same, except that the probabilities for each subsequent transition state differ. We can assume two distinct states s′ and s′′ where the probabilities diverge. Let p(s′ ), p(s′′ ) and ˆ p(s′ ), ˆ p(s′′ ) denote the transition probabilities for a1 and a2, respectively, while all other probabilities (denoted collectively as C remain identical). Assume p(s′ ) < ˆ p(s′ ) and p(s′′ ) > ˆ p(s′′ ). We then have

By Lemma 4.1, to prove that the entropy decreases, it suffices to show the existence of α(s′ ) and α(s′′ ) such that

To prove this inequality, we reformulate it as

Simplifying further yields

Apparently, there exist α(s′ ) and α(s′′ ) that satisfies Qtotal(s, a1) = Qexploit(s, a1) + ˆ δ(s, a1) and Qtotal(s, a2) = Qexploit(s, a2). This means H(πexploit|s) > H(πtotal|s), i.e., the probability of optimal policy increases while entropy starts to decrease. In this scenario, the degree of exploitation adaptively increases while the level of exploration decreases.

Case III - Exploitation Dominant: for any s, when α(s) ≡ 1, we have Qtotal(s, a1) = Qexploit(s, a1), and Qtotal(s, a2) = Qexploit(s, a2). This means the entropy remains unchanged, i.e., H(πexploit|s) =H(πtotal|s). In this scenario, the strength of exploration is zeroed out and exploitation is dominant.

From the above theorem, we know that as α(s) changes, the entropy of our policy can get higher or lower than that of πexploit. This implies that our method can conduct a dynamic level of exploration and adaptively alternate between exploration and exploitation. Visual evidence is provided in Fig. 3(b). Our design also aligns with the principle of Occam's Razor [Murphy, 2012; Domingos, 2012] in that exploration rewards should not be posited without necessity.

Guarantee of Entropy Minimization. We further prove that based on Eqn. (4), the entropy of policy is guaranteed to decrease in the global perspective.

Theorem 4.2. (Decay of entropy after optimization.) During the optimization, the sequences {ˆ δk-1(s, ai)}Ni=1 at step k and the sequences {ˆ δk(s, ai)}Ni=1 at step k + 1 satisfy the conditions of Lemma 4.1, in addition to the following relation, for i ≤ j:

The proof of Theorem 4.2 can be obtained by applying Lemma 4.1 and is shown in Appendix. The steady decrease of entropy is also validated through visualization of entropy curves in Fig. 3. Note that Theorem 4.2 applies to a broad spectrum of RL methods with intrinsic motivation that approaches zero during training.

5 Validation of Adaptive Theory: AdaZero

Inspired by the above theoretical findings, we formalize an exploration-exploitation dynamic adaptive framework named AdaZero. As illustrated in Fig. 1, AdaZero consists of three parts: a state autoencoder, a mastery evaluation network, and an adaptive policy optimization mechanism.

State Autoencoder. The state autoencoder aims to motivate the agent's exploratory behavior of searching for unknown policies to get familiar with the environment. As shown in Fig. 1A, the state autoencoder receives raw states s via the interaction between the agent and the environment and learns to reconstruct the input images ˆ s = gθ(s), where θ is the network parameters. The reconstruction error Rexplore(s, a) is defined as follows:

We use the reconstruction error to drive exploration. A larger error indicates insufficient network training on the state, and hence, exploration needs to be enhanced for this state. Conversely, a smaller error indicates that the training on that state has been sufficient, leading to an automatic decrease in exploratory rewards.

Mastery EvaluationNetwork. The above state autoencoder provides information about where to explore. Furthermore, determining whether or not and how much to explore based on the agent's mastery level of the states is also crucial. To this end, AdaZero involves a mastery evaluation network to provide real-time assessment of the necessity of exploration, as shown in Fig. 1B.

The mastery network is implemented as a binary classifier denoted as fω. At the training stage, the network receives mixed data of real and fake state images reconstructed by the state autoencoder, denoted as s and ˆ s, respectively. It is trained to distinguish between the two kinds of inputs and output the probability of the inputs being real images, with a cross-entropy loss defined as follows:

where ω is the network's parameters, S and ˆ S are the set of real state images and reconstructed images, respectively.

At the inference stage, the evaluation network only receives the reconstructed images from the state autoencoder and outputs the probability α(ˆ s) ≜ fω(ˆ s). Hence, α(ˆ s) ∈ [0, 1] can naturally serve as a measure of AdaZero's grasp of the current training state information.

Adaptive Mechanisms. Inspired by Eqn. 4 and Theorem 4.1, by integrating the reconstruction error Rexplore provided by the state autoencoder and the estimated level of mastery of the state information α(ˆ s) from the evaluation network, we implement the adaptive mechanism of AdaZero depicted in Fig. 1C through the following formula:

When the agent reaches a high level of mastery in the current state, the exploration intensity should be reduced to increase the exploitation ratio. Conversely, when the level of mastery is Return TimeSteps(10Million)

0.8

1.0

PPO

Method

AdaZero_PPO

Montezuma's Revenge

Return

6630

PPO

440

AdaZero_RND

6390

RND

AdaZero_RPO

1810

6153

RPO

0

(a) Integration Experiment

(50Million)

(b) Comparative Experiment

Figure 2: Main experiments in the most challenging Montezuma's Revenge. (a) We integrate AdaZero into three representative RL methods

and show that AdaZero can bring improvements; (b) Our method also presents an advantageous performance compared with other advanced

baselines. Equipped with AdaZero, RND reached the score published in the original paper within only one-tenth of the training steps.

(1) HumanoidStandup

(2) Swimmer

0.2 0.4

0.6 0.8 1.0

(3) Reacher

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

(4) Walker2d

0.4

AdaZero_PPO

0.6

0.4

0.6

0.8

1.0

RND

TRPO

(a) Rewards in MuJoCo.

low, the exploration intensity should be increased to encourage the agent to better learn from the various information about the state. As α(ˆ s) ∈ [0, 1] reflects the agent's ability to grasp valuable information in the reconstructed states, we use (1 - α(ˆ s)) to control the strength of intrinsic rewards adaptively, hence dynamically balance the exploration and exploitation.

Compared with methods that decay intrinsic rewards [Badia et al., 2020], the adaptive mechanism of AdaZero is fulfilled by end-to-end networks. According to Theorem 4.1, from a local perspective, the exploratory incentive of AdaZero can be adaptively increased or decreased, or completely eliminated in response to the training progress and environment changes. Furthermore, according to Theorem 4.2, the entropy of AdaZero decreases globally, thereby theoretically guaranteeing the eventual transition from exploration to exploitation.

6 Experiments

6.1 Main Experiments

To provide empirical support for our theoretical findings and validate the effectiveness of AdaZero framework, we conduct experiments on a variety of RL tasks, ranging from dense to sparse reward environments, continuous to discrete action spaces, and quantitative comparisons to visual analyses.

We compare AdaZero with the recent competitive and widely-used RL algorithms, including PPO [Schulman et al., 2017], RND [Burda et al., 2018], and RPO [Gan et al., 2024]. For continuous tasks, we additionally include TRPO [Schulman et al., 2015] as our baseline. As AdaZero is designed to be flexible and can be easily integrated into existing algorithms, we first compare the performance of the baselines integrated with AdaZero versus without AdaZero. Then, we focus on PPO with AdaZero as a representative of our method.

Standarad

deciation

＋

－

＋

－

＋

－

＋

－

＋

－

＋

－

52.6

0

207.1

452.2

273.7

0

160

140

120

100

80

60

40

20

4000

3000

2000

1000

In the remainder of this paper, AdaZero refers to PPO integrated with AdaZero. Note that RND integrated with AdaZero (AdaZero RND) is implemented by using RND's intrinsic reward module and adapting its magnitude through the weights predicted by AdaZero's autoencoder and evaluation network.

We conducted integration experiments in the most challenging RL environment, Montezuma's Revenge [Bellemare et al., 2016]. The results are shown in Fig. 2. Overall, AdaZero boosts the total returns up to about fifteen times for PPO and 3.5 times for RND in Fig. 2 (a). Besides, AdaZero significantly outperforms other baselines in Fig. 2 (b). It demonstrates that within the same training steps, AdaZero surpasses other algorithms in final returns.

The integration experiment results in Fig. 2 (a) also confirm AdaZero's effectiveness in collaboration with different methods and in highly challenging environments. Compared with algorithms like PPO and RPO, which primarily focus on exploitation, AdaZero maintains a high exploitation rate and supports selective exploration. Compared with the explorationcentric RND, AdaZero enhances its exploitation ability. Moreover, it can be observed that there are more horizontal fluctuations in the three curves corresponding to AdaZero. This indicates that our dynamic adaptive mechanism enables the agent to discover more localized policies during training, thereby enhancing the overall training efficiency. Without implementing the proposed AdaZero, PPO and RPO would be trapped in invalid policies and almost fail.

6.2 Generalization Experiments

To demonstrate AdaZero's robustness and broad applicability, this subsection presents its performance on extensive environments [Taiga et al., 2020], including other Atari games as well as continuous-space environments of MuJoCo. Fig. 4 indicates that AdaZero is robust in challenging discrete-space tasks. Fig. 3 (a) shows the performance of AdaZero in continuousaction tasks. The results indicate that our algorithm outperforms baseline algorithms in MoJuCo environments, confirming that AdaZero is broadly effective in continuous spaces.

6.3 Computational Cost and Efficiency

Our exploration network size is similar to that of RND. RND requires two encoders of equal size ( 2M x 2 parameters) for exploration reward evaluation, while our exploration network ranges from 2 to 4M (size varies with feature layer). The evaluation network is a convolutional network with 0.5M parameters. We recorded the total training time across all environments. Compared to RND, our method requires about 1.3 times the time to reach the same total steps, mainly due to the evaluation network's involvement in training. However, our method achieves the same score in less time, showing higher time efficiency. In Atari, we recorded an average improvement of about 43% in time efficiency at the same score. Time efficiency is calculated as: Time efficiency = tRND-tAdaZero tRND . The results are shown in Tab. 1.

7 Conclusion

This paper reveals the role of entropy in policy optimization. Based on this motivation, we re-examined the explorationexploitation dilemma from an entropy perspective and proposed an adaptive exploration-exploitation mechanism. Inspired by this mechanism, we formalized AdaZero, which adapts using end-to-end networks. Both theoretical analysis and experimental results demonstrate that our method enables selective exploration while maintaining effective exploitation, outperforming traditional RL algorithms across diverse environments. In future work, we will extend the adaptive theory beyond decision-making tasks.

Acknowledgements

We would like to thank all the reviewers for their constructive comments. This work was supported by the NSFC (62341407, U24B20166, 62495102), Beijing Natural Science Foundation (Grant L223004), and "111" Project (B18001).