What Hides behind Unfairness? Exploring Dynamics Fairness in Reinforcement Learning

1 Introduction

Algorithmic fairness has emerged as an increasingly important research topic in the era of autonomous decision-making systems driven by machine learning [Kasy and Abebe, 2021; Mehrabi et al., 2021; Pessach and Shmueli, 2023]. The widespread application of machine learning algorithms in fields like education [Wolff et al., 2013; Anderson et al., 2019], finance [Dwork et al., 2012; Lee et al., 2014], and law [Angwin et al., ; Tolan et al., 2019] carries the risk of unintentionally perpetuating or introducing biases and discrimination against race, gender, or other types of sensitive attributes. Despite numerous approaches developed to address algorithmic fairness, many focus on supervised learning or decision-making problems without state transitions [Hardt et al., 2016; Zhang et al., 2017; Liu et al., 2018; Patil et al., 2021]. However, fairness is not static [D'Amour et al., 2020]. In reality, the dynamic nature of the world requires intelligent agents to trace the ongoing interaction with the external environment and adjust their behavior based on environmental feedback. This highlights the need to study long-term fairness in sequential decision-making problems.

Recent works have proposed to formulate and investigate long-term fairness within the framework of Markov Decision Processes (MDPs) [Wen et al., 2021; Chi et al., 2022; Tang et al., 2023], which makes this topic closely related to reinforcement learning. Under this framework, a group fairness problem can be formulated as follows: different demographic groups are represented by a set of MDPs that share the same state and action space and run in parallel, and the agent's decisions trigger group-level state transitions and reward assignments. In this case, long-term fairness regarding the well-being [Plecko and Bareinboim, 2022] of each group can be measured by the difference in future returns under the agent-specified policy. Some other fairness notions have been proposed as well [Liu et al., 2018; Mouzannar et al., 2019; Zhang et al., 2020; Tang et al., 2023], yet the sources of inequality that lead to unfairness remain unclear.

To highlight the importance of addressing this gap, consider an illustrative example involving two demographic groups, in which the state of each group is represented by a one-dimensional variable, as shown in Figure 1. The first two columns in this figure show the possible outcomes when two common fairness notions are adopted as criteria. The left one corresponds to fair decision, requiring the agent to make consistent decisions for both groups, regardless of sensitive attributes. This criterion is popular because people usually expect decision-makers to provide equal resources and opportunities for different groups instead of treating them disparately, but it does not guarantee fair long-term well-being. The middle column, on the other hand, pursues fair state, urging the agent to improve the socio-economic status or qualification profile of the disadvantaged group, so that different groups will reach similar states in the long run. While this notion is very tempting, its implication for long-termwell-being is also uncertain, as the disparities would remain if the reward assignment mechanism governing the environment is biased against the disadvantaged group. Both criteria may lead to different outcomes depending on the environmental dynamics, requiring us to investigate the mechanisms governing the environment, which is overlooked in previous studies.

In this paper, we address the above challenge from a causal perspective. Specifically, we trace the sources of inequality by decomposing the causal effect of sensitive attributes on long-term well-being. We formulate the long-term fairness problem via MDP and introduce a novel notion called dynamics fairness to capture the fairness of the mechanisms that govern the environment, measuring the expected changes in the next state and the reward induced by changing the identity of a demographic group (the value of a sensitive attribute) while holding everything else constant. This notion provides a useful tool for analysing the sources of inequality, distinguishing the inequality introduced by the environmental dynamics from those induced by decision-making or inherited from the past. Drawing inspiration from research on mediation analysis in the field of causal inference [Pearl, 2009; Pearl et al., 2016; Pearl, 2022], we propose to characterize dynamics fairness by natural direct effects. Since the computation of these effects involves nested counterfactuals that cannot be observed in the factual world, we further derive identification formulas that allow us to reliably estimate these quantities from data. As depicted by the right column of Figure 1, breaching the fair decision and state criteria is generally inevitable when facing an environment that violates dynamics fairness. In this case, to achieve fair long-termwell-being, an agent must adopt compensatory policies for the disadvantaged group. Conversely, when the environment satisfies dynamics fairness, these criteria can be aligned after eliminating the inequality inherited from the past, as indicated by the star.

In summary, our main contributions are as follows:

• We introduce a novel causal fairness notion called dynamics fairness. Previous notions were mainly defined in terms of observable disparities in states, decisions, or

rewards. To the best of our knowledge, dynamics fairness is the first notion that is defined on the underlying mechanisms governing the environment. Thus, it fills in the missing piece in studying long-term fairness in reinforcement learning.

• We further derive a method to quantitatively evaluate dynamics fairness through theoretical analysis, which allows this notion to be applied in practical situations. More importantly, this is a general method that does not rely on making parametric assumptions about the environment, such as linearity.
• We conducted extensive experiments to verify the theoretical results and demonstrate the effectiveness of the proposed methods in explaining, detecting, and reducing inequality in reinforcement learning problems.

1.1 Preliminaries

In this paper, we apply causal inference techniques to explore the mechanisms that explain the inequality in reinforcement learning. We adopt the framework of structural causal models (SCMs) [Pearl, 2009; Pearl et al., 2016], which provide a mathematical formalization of causality. SCMs, denoted as C := ⟨V, U, F, P(u)⟩, consist of endogenous (observable) variables V and background (latent) variables U, linked by structural equations fVi in F. These equations model the causal mechanisms among variables, relating each Vi ∈ V to its parents pa(Vi) and associated exogenous variables UVi , under a probability distribution P(u) over U. An SCM induces a causal diagram G over the nodes V, in which a directed edge Vi → Vj exists if Vi is an argument of fVj .

SCMs provide a formal language for articulating counterfactual statements, allowing researchers to examine hypothetical scenarios by evaluating potential outcomes under varied situations. Specifically, researchers use Yu(x) to denote the potential outcome of a variable Y obtained by solving the equations in F with X = x for an individual U = u. In causal inference, many research efforts focus on predicting causal effects such as the total effect (TE) TEx,x′(Y ) := E[Y (x′ )] - E[Y (x)], where X is the treatment variable and Y is the outcome variable. TE measures the difference between the potential outcomes of a variable under different treatments for the whole population (rather than for a specific individual u). In this paper, we are interested in the TE of the sensitive attribute on the future return, i.e., how a change in identity would affect a demographic group's long-term wellbeing if all other factors were held constant? 1 The following section explores decomposing this effect into distinct components, examining the sources of inequality in RL problems, particularly focusing on the role of environmental dynamics.

2 Analysing Inequality in Reinforcement Learning

In this section, we aim to explore and explain the inequality in reinforcement learning through a causal lens. We first present the problem formulation, employing causal modeling to analyse environmental dynamics in sequential decision-making scenarios. Next, we conduct an in-depth analysis of inequality and investigate its sources through the decomposition of inequality. Furthermore, we introduce dynamics fairness and propose an approach for evaluating this counterfactual concept quantitatively. Lastly, we present a model-based reinforcement learning algorithm that incorporates dynamics fairness into the planning process.

2.1 ProblemFormulation

Problem Setup In reinforcement learning, sequential decision-making problems under uncertainty are often modeled as an MDP M = (S, A, P, R, µ, γ). S and A denote the state and the action space respectively; P : S × A → ∆S is the transition kernel, where P(s′ |s, a) is the probability of transitioning to state s′ after taking action a in state s; R : S × A → R specifies the immediate reward received after taking action a in state s; µ ∈ ∆S stands for the initial state distribution. At step t, the goal of an RL agent is to maximize the future return Gt = Rt + γRt+1 + γ 2Rt+2 + · · · = ∑∞k=0 γ kRt+k, where γ ∈ [0, 1) is the discount factor. In model-based RL, a common practice is to approximate the real environment using a parametric model fθ, which is employed to generate additional training data for policy learning or predict future transitions and rewards for planning. To model the environmental dynamics of multiple demographic groups, we consider N MDPs, all sharing the same state and action space, with each MDP representing a distinct demographic group. We abstract the group identity as a variable Z, referred to as the sensitive attribute. Then we can characterize the set of all N MDPs through an augmented dynamics model f ∗ θ that takes Z as an additional input. This is a common strategy used to address the dynamics generalization problem [Lee et al., 2020; Guo et al., 2022], enabling agents to learn the environment efficiently and accurately.

1 While in reality manipulating one's identity is generally infeasible, one can still obtain reliable estimates of the causal effect of interest under appropriate structural assumptions [Pearl, 2009].

SCM Representation of The Dynamics Model Based on the above formulation, we can cast the augmented dynamics model into an SCM C using auto-regressive uniformization (See Lemma 2 in [Buesing et al., 2019]). For example, let's consider the conditional distribution P(R|S,A,Z) and denote its cumulative distribution function as F(R|S,A,Z). We can construct a new variable ¯R := F -1 (UR|S,A,Z) where UR is a uniform random variable sampled from [0, 1]. Since the distribution of ¯R matches that of R, by defining fR := F -1 (US′ |S,A,Z), we obtain the structural equation for R. A similar approach can be applied to other variables as well. The causal diagram G induced by the resulting SCM C is illustrated in Figure 2. Note that the impact of the sensitive attribute may cover various facets of the system, including state transitions, decision-making, and reward assignments. Therefore, inequality may stem from multiple sources following different causal paths in G.

2.2 Decomposition of Inequality

In this work, we focus on the inequality in long-term wellbeing, a causal quantity that is captured by the difference in future return Gt when setting the sensitive attribute Z = z1 compared to Z = z0. Using the causal language, we can now formally define this quantity as follows.

Definition 1 (Well-being Gap). At step t, the well-being gap between the two demographic groups distinguished by the sensitive attribute Z is defined as:

This quantity measures the total effect of the sensitive attribute Z on the future return Gt. Since Gt is a linear combination of the reward signals, the well-being gap satisfies the following lemma:

Lemma 1. The well-being gap TEz0,z1 (Gt) can be decomposed into the sum of reward gaps at each time step:

where TEz0,z1 (Rt) := E[Rt(z1)] - E[Rt(z0)] is the reward gap at step t.

Lemma 1 indicates that we can focus on the reward gap at each time step to examine the sources of inequality in long-term well-being. In order to analyse the reward gap TEz0,z1 (Rt) at step t, we present the local causal diagram in Figure 3 (Left). From the diagram, we can see that TEz0,z1 (Rt) is affected by the sensitive attribute Z through two types of causal paths: the direct path Z → Rt and the indirect paths mediated by the current state St or the decision At. Notably, Z → St carries the inequality inherited from historical factors as well as the one directly caused by Z.

Next, we introduce two counterfactual quantities that are useful for a refined decomposition of the reward gap. We omit the subscript t when the context is clear.

Definition 2 (Natural Direct and Indirect Effect). The natural direct and indirect effects of sensitive attribute Z on reward R are defined, respectively, as:

Figure 3 (Right) illustrates these two quantities, both involving nested counterfactuals. For instance, in NDEz0,z1 (R), the first factor involves computing the expected reward when S and A are set to the values they would have naturally taken under Z = z0 while Z is set to z1. Perhaps surprisingly, despite being not realizable in the factual world, these quantities are of great interest and are widely used in many practical applications. In discrimination cases, by keeping the state S and decision A to the level they would have naturally attained under Z = z0, NDEz0,z1 (R) measures the advantage that the individuals in the disadvantaged group Z = z0 would have gained if there were no direct discrimination within the reward assignment process. Conversely, NIEz1,z0 (R) measures the advantage that the individuals in the disadvantaged group Z = z0 lose due to indirect discrimination.

Based on Lemma 1 and the above definitions, we can now present a quantitative decomposition of the well-being gap:

Theorem 1 (Causal Decomposition of Well-being Gap). The well-being gap TEz0,z1 (Gt) can be decomposed as follows:

Theorem 1 offers a general decomposition of the wellbeing gap, quantitatively explaining how it is accumulated over time, without assuming any specific functional form of the structural equations in C. The following lemma further reveals the qualitative relationship between the natural direct and indirect effects and the causal paths.

Lemma 2. Consider a set of MDPs described by the augmented dynamics model f ∗ θ , along with the induced causal diagramG, the following statements hold: If NDEz0,z1 (R) ̸= 0, then there exists a direct path from Z to R; Similarly, if NIEz1,z0 (R) ̸= 0, then there exist indirect paths connecting Z and R.

That is to say, the natural direct and indirect effects provide hints for structural information about the causal diagram, thus helping us identify the causal paths that are responsible for the gap. Summing up, the analysis presented herein systematically examines the sources of inequality through a causal lens. By introducing the natural direct and indirect effects, we provide both quantitative and qualitative explanations of the well-being gap, offering insights into the intricacies of inequality in reinforcement learning problems.

2.3 Dynamics Fairness

Beyond merely explaining the sources of inequality, in practice, we also seek clarity regarding the responsibility for such inequality. Theorem 1 and Lemma 2 demonstrate that the inequality accumulated at each step can be attributed to two types of causal paths: the direct path (Z → R) and the indirect paths (Z → S → R, Z → A → R, and Z → S → A → R). However, the inequality introduced by the environmental dynamics and the decisions is still coupled together. In other words, NIEz0,z1 (R) ̸= 0 can not tell us how to determine responsibility between the two. Before diving into further details, we first formally define the notion of dynamics fairness.

Definition 3 (Dynamics Fairness). The dynamics is fair regarding the sensitive attribute Z if there are no direct paths from Z to either the reward R or the next stateS′ in the causal diagram G induced by the augmented dynamics model f ∗ θ .

Definition 3 provides a structural condition for the environment to satisfy dynamics fairness. Banning the direct paths from the sensitive attribute to the reward and the next state ensures inequality is not introduced by the environmental dynamics. As a result, there are only two possible sources of inequality: historical inequality, i.e., the disparity in the current state of different demographic groups, and the inequality introduced by the decisions, i.e., disparate treatment in the decision-making process. Specifically, the historical inequality can be estimated by the observed disparity in the current state E[S|Z = z1] - E[S|Z = z0], which is easy to measure. If the environment satisfies dynamics fairness and there is no historical inequality, then the well-being gap is completely attributed to the decisions. In this way, we can solve the problem of attributing responsibility for inequality. Building upon Lemma 2, we establish the following theorem:

Theorem 2 (Criterion for Dynamics Fairness Violation). The environment is considered to violate dynamics fairness if either NDEz0, z1(R) ̸= 0 or NDEz0, z1(S′ ) ̸= 0 holds.

Theorem 2 offers a quantitative criterion for dynamics fairness. By checking the natural direct effects of Z on reward R and next state S′ , we can determine whether the mechanisms governing the environmental dynamics would introduce inequality during the sequential decision-making process, thus contributing to the well-being gap in the long run. More importantly, it also provides mathematical justification for compensatory policies. If the environment violates dynamics fairness, then disparate treatment is necessary to compensate the disadvantaged group when aiming to achieve equality in long-term well-being. On the other hand, if the environment is fair, then disparate treatment should be limited to a minimum level, not only for the sake of fair decision-making but also to avoid widening the well-being gap.

As mentioned earlier, the computation of NDEz0,z1 (·) involves nested counterfactuals, which require data that we cannot obtain in the factual world. Fortunately, under the structural assumptions implied by the augmented dynamics model, we can derive identification equations for these effects.

Theorem 3 (Identification of Dynamics Fairness). Under the structural assumptions implied by the augmented dynamics model depicted by the local causal diagram, the natural direct effects of Z on R and S′ can be identified as follows:

In the above equations, all quantities on the right-hand side are expressed using conditional expectations or probabilities rather than counterfactuals, and they can be estimated directly from data using standard statistical methods. As a result, Theorem 3 provides a practical way to determine whether the environment satisfies dynamics fairness. Proofs are included in Appendix A in the supplementary material.

2.4 Algorithm for Achieving Long-TermFairness

So far, we have examined the sources of inequality and the associated responsibilities, with a particular focus on the influence of environmental dynamics. The presented results not only demystify the intricate relationships behind unfairness in RL problems but also provide valuable guidance for designing RL algorithms to tackle inequality. Based on the above analyses, we propose a model-based RL algorithm named InsightFair that incorporates dynamics fairness into the planning process. Specifically, the agent iteratively collects environmental data and trains an ensemble augmented dynamics model f ∗ θ , which is later used to simulate the environment and evaluate the action sequences for planning. At each epoch, the agent first evaluates dynamics fairness following Theorem 3 using f ∗ θ and the collected data. If the environment violates dynamics fairness, then disparate treatment is considered necessary in the planning process. Otherwise, the agent samples consistent action sequences for all groups when the observed disparity in the current state approaches zero. The optimization objective is regularized by a constraint on the estimated well-being gap to induce a fair future. The detailed algorithm is presented in Appendix B.

3 Experiments

In this section, we empirically validate the theoretical results of our work through extensive experiments. Specifically, the results presented here aim to answer the following questions: (1) Does the proposed decomposition offer an accurate quantitative explanation for inequality? (2) Can the proposed identification formula reliably detect cases where the environment violates dynamics fairness? (3) Does the proposed algorithm, InsightFair, work effectively with the aid of dynamics fairness? To be self-contained, we include additional details and results in Appendix C in the supplementary material.

3.1 Explaining Inequality

To answer the first question, we consider a simple non-linear model for the reward function, i.e., R = 1 if w0+w1z+w2s+ w3a + uR ≥ 0 and R = 0 otherwise, where w0, w1, w2, w3 are the model parameters and UR ∼ logistic(0, 1) is the background variable. Since the reward function is fixed within an episode and the return is a linear combination of the rewards, we can examine the well-being gap by analysing the decomposition of the reward gap TEz0,z1 (R).

Figure 4 illustrates the comparison between the total effect TEz0,z1 (R) and the natural direct and indirect effects NDEz0,z1 (R) and NIEz1,z0 (R), respectively. The values are computed analytically under two sets of model parameters w1 = w2 = w3 = 0.1 and w1 = w2 = w3 = 3.0 along with w0 varying in the range [-2.5, 2.5]. First, we can see that the sum of NDEz0,z1 (R) and -NIEz1,z0 (R) accurately explains the reward gap TEz0,z1 (R) in both settings. Second, by varying w0, the proportion of the direct and indirect effects changes accordingly. In the right figure, the dominance of the direct effect suggests that the inequality is mainly introduced by the environmental dynamics instead of other factors.

3.2 Detecting Violation of Dynamics Fairness

In this part, we empirically verify the effectiveness of the proposed identification formula by customizing the ML-fairnessgym suite [D'Amour et al., 2020] for group fairness problems. Specifically, we use the Allocation environment that contains two demographic groups. At each step, some incidents like rat infestations occur in both groups, depending on their current state. The agent must allocate resources effectively to minimize both missed incidents and resources allocated. Groups do not interact with each other. By manipulating the reward function and the transition dynamics, we can generate different environments to simulate scenarios where the environment satisfies or violates dynamics fairness.

We conduct two sets of experiments to evaluate NDEz0,z1 (R) and NDEz0,z1 (S′ ), respectively. In the first set, we vary the reward function while keeping the transition dynamics fixed. Two parameters β0 and β1 are used to control the relative advantage of the second demographic group. When β1 > β0, the second group is advantaged, and vice versa. The environment is considered fair when β1 = β2, i.e., the relative advantage reduces to zero. The second set of experiments is conducted similarly, except that we fix the reward function and vary the parameters controlling the transition dynamics to determine the relative advantage. For both sets, we conduct a total of 64 experiments by varying the parameters in a fixed range to generate a series of environments and estimating the natural direct effects using the proposed identification formula. The results are shown in Figure 5. From this figure, it is clear that the natural direct effects approach zero when the environment satisfies dynamics fairness (as colored in white on the diagonal). On the other hand, when the environment violates dynamics fairness, the natural direct effects vary with the magnitude of the relative advantage, as indicated by the darker colors in the heatmaps. These results empirically support that the proposed approach can reliably detect violations of dynamics fairness.

3.3 Promoting Long-TermFairness

To answer the third question, we evaluate the performance of the proposed algorithm by comparing it with the base algorithm, PETS [Chua et al., 2018], which is a popular modelbased RL algorithm but does not consider fairness. In addition, we consider two variants of PETS. The first variant, Fair-A, requires the agent to satisfy the fair decision criterion, sampling consistent action sequences for all groups. The second variant, Fair-S, incorporates a constraint regarding state disparities into the optimization objective, aiming to achieve equality in the state for all groups. Existing studies [Wen and Topcu, 2018; Liu et al., 2021] have shown that model-based algorithms can effectively satisfy constraints while maintaining good task performance. Note that our focus in this paper is orthogonal to these works. The experiments presented herein aim to examine the effectiveness of the proposed fairness notion when incorporated into a model-based RL algorithm, particularly in promoting fair long-term wellbeing when the environment violates dynamics fairness and in aligning different criteria in a fair environment.

The learning curves in Figure 6 (Left) illustrate the average return and the well-being gap for all algorithms in the Allocation environment. The results are averaged over five runs with different random seeds. InsightFair exhibits a slightly lower task performance compared to PETS and Fair-A, while considerably reducing the well-being gap, indicating its effectiveness in promoting long-term fairness when the environment violates dynamics fairness. Fair-S also reduces the well-being gap but at the cost of task performance, as it sacrifices the utility of the advantaged group to achieve equality in the state. In the next experiment shown in Figure 6 (Right), we consider a fair environment where the relative advantage is zero but the advantaged group begins with a better initial state. We visualize the changes in state and decision gap (the absolute difference in actions) between the two groups. InsightFair makes consistent decisions for both groups when state disparities approach zero, resulting in zero decision gap in most steps, while PETS demonstrates disparate treatment throughout the episode. These results suggest that InsightFair effectively aligns the fair state, action, and well-being criteria in a fair environment.

4 RelatedWork

Long-term Fairness Consideration of fairness in machine learning systems has evolved beyond static settings to a dynamic perspective [D'Amour et al., 2020], i.e., autonomous agents may interact with the environment over time and have a long-term impact on fairness. Notably, growing evidence [Liu et al., 2018; Zhang et al., 2020] support concerns about static fairness measures, as they may jeopardize long-term fairness. In recent years, there has been a considerable amount of work devoted to promoting longterm fairness [Mouzannar et al., 2019; Wen et al., 2021; Puranik et al., 2022; Chi et al., 2022; Tan et al., 2022; Yu et al., 2022; Xu et al., 2023]. [Wen et al., 2021] investigated the long-term fairness problem under scenarios in which individuals receive decisions repeatedly. Specifically, they define fairness as the gap between the expected cumulative rewards of individuals in different groups. The proposed algorithm relies on a known model and can be reduced to solving a linear programming problem in the finite-state MDPs setting. [Chi et al., 2022] adopt a similar definition of fairness. Their algorithm employs integral probability metrics to align the state visitation distributions among groups, thereby mitigating return disparity. More recently, [Yu et al., 2022] proposed a policy optimization algorithm that incorporates fairness constraints into the advantage function. This enables an agent to efficiently explore the environment and learn a policy that strikes the balance between utility and fairness. However, [Xu et al., 2023] noted that assuming the fairness metric as part of the state is generally impractical because long-term fairness necessitates the agent to focus on the future rather than the past. Their work studied supply-demand MDPs, a variant of MDPs that contain two time-varying variables, supply and demand, for each group depending on states and actions. The authors suggested using the ratio between the cumulative group supply and demand to measure long-term well-being and proposed a policy optimization algorithm that can effectively reduce the discrepancy in this ratio. Additionally, multi-objective RL algorithms [Reymond et al., 2022] have shown potenial in handling the complex trade-offs inherent in balancing various fairness criteria. In this paper, we systematically examine long-term fairness for demographic groups characterized by a set of different MDPs. Our work features causal modeling of the agent-environment interaction and aims to identify the sources of inequality, thereby providing a new perspective on studying the long-term fairness problem.

Causal Fairness Analysis Causal fairness analysis, an emerging trend in fairness research, leverages techniques and theories from the causal inference community to advance the exploration of fairness problems in machine learning [Creager et al., 2020; Plecko and Bareinboim, 2022; Deng et al., 2023]. This methodology not only rigorously defines fairness but also provides effective and reliable solutions to ensure it." For example, [Kusner et al., 2017] proposed counterfactual fairness for fair prediction problems, requiring the distribution over possible predictions of an individual to remain unchanged in a world where its sensitive attributes had been changed. In fact, counterfactuals provide a versatile framework for formulating various fairness notions of different granularity. [Zhang and Bareinboim, 2018] introduced different types of counterfactual effects for the fair decision problem. These effects offer fairness notions at the group level, helping researchers interpret various discrimination mechanisms. Furthermore, [Nabi et al., 2019] and [Wu et al., 2019] demonstrated how causal graphs can assist researchers in defining fine-grained fairness associated with specific causal paths. For instance, using sensitive attributes or their proxies is deemed reasonable in some cases due to business necessity [Plecko and Bareinboim, 2022] but may lead to discrimination in other cases. Recent research has extended the study of causal fairness to contextual bandits and outcome control problems [Huang et al., 2022; Plecko and Bareinboim, 2023]. Unlike previous studies, our paper delves into the fairness problem in the sequential decision-making setting, introducing a novel causal fairness notion, dynamics fairness, which captures the fairness of the mechanisms governing the environment.

5 Conclusion

In this paper, we systematically studied inequality in RL problems through a causal lens, aiming to demystify the intricate relationships behind unfairness. We introduced a causal model to formalize the agent-environment interaction and provided a quantitative decomposition of the well-being gap, which explains how inequality is accumulated over time. Furthermore, we proposed dynamics fairness, a novel causal fairness notion that, for the first time, captures the fairness of the underlying mechanisms governing the environment. To measure this counterfactual quantity, we proposed identification formulas that do not rely on specific functional forms or a known model. Extensive experiments were conducted to verify the theoretical results and demonstrate the effectiveness of the proposed approaches. In the future, we plan to extend our work to more complicated scenarios, such as partially observable and non-stationary environments.