Unintended Consequences: Updating Causal Models

1 Introduction

Causal reasoning is difficult; agents routinely misunderstand, misinterpret, or simply overlook the true causal relationships between variables of interest. Examples of such erroneous judgments are plentiful, ranging from the pedestrian (superstitiously carrying a lucky charm) to the existential (misjudging humanity's effect on global climate). Because an agent will decide which actions to take based on her understanding of the effect of her actions, errors in causal reasoning are costly, leading to suboptimal or inefficient choices. An agent may entertain different causal models, each reflecting a different understanding of the causal relationships, and hold probabilistic beliefs over these. In dynamic or repeated environments, an agent who entertains incorrect causal judgements might observe evidence that contradicts her initial understanding, requiring her to re-evaluate her beliefs. However, as we show, this re-evaluation may not result in correct beliefs or stable behavior, even after an infinite number of interactions.

In this paper, we examine more generally how causal reasoning affects an agent's choices and how feedback on those choices leads to updated causal beliefs. In order to formalize causal reasoning, we draw on the literature on structural causal models `a la [Pearl, 2000]. We first examine the general problem of updating causal beliefs in the face of novel (and possibly inexplicable) data. We model an agent who is uncertain of the true causal model, and therefore entertains a probabilistic belief over the set of possible models. To model the agent's reaction to inexplicable evidence (that is, when she observes outcomes that cannot be explained by any model in the support of her beliefs), we endow the agent with a conditional probability system (cps) [Popper, 1934; Popper, 1968; de Finetti, 1936], encoding the agent's behavior conditional on any (including potentially probability 0) observation. 1

We then consider how causal beliefs influence choices by building a model of agency and utility on top of the usual

take actions that consist of intervening on a set of variables and making partial observations about the outcome of the intervention. The agent derives utility as a function of the values of the variables, and thus her ability to intervene has a direct consequence on her utility. In dynamic environments, the agent also derives value from her observation regarding the effect of interventions, as such information forms the basis of belief updating, and therefore allows for better future decision making.

Using these two components, we propose a notion of steady state, where the feedback received from an agent's optimal action, given her current beliefs about the true causal model, can be rationalized by those beliefs. In such a case, the optimal action does not induce the agent to revise her beliefs, and therefore, the action remains optimal. To get a sense of the relation between an agent's choice of action, the data that is generated, and the re-evaluation of beliefs, consider the following two examples. In both, the protagonist takes the incorrect action (as judged from the modeler's perspective) on the basis of their faulty causal reasoning; the difference lies in how the agents react to the observed evidence:

Example 1. During British East India Company rule of colonial India, to combat the proliferation of dangerous serpents, the Governor came up with what appeared an ingenious solution: pay a bounty to any man who brought in the head of a dead cobra. This "cash for snakes" program worked for a while, but was followed by a marked uptick in the number of cobras plaguing the city. Indeed, entrepreneurial locals began breeding snakes for the express purpose of collecting the bounty, but, owing to lax security, many snakes escaped into the wild, dramatically growing the population.

1 We interpret the beliefs conditional on probability 0 events as the analyst's model of the agent's behavior, and not as beliefs consciously held by the agent. In other words, the agent need not anticipate how she will react to inexplicable evidence.

The Governor learns that his action was misguided, since the observation that arises subsequent to his choice of action cannot be rationalized by his initial understanding. Our second example, by contrast, shows that an agent can remain in perpetual delusion, if the evidence generated by her action accords perfectly with her beliefs, despite the fact that those beliefs are incorrect.

Example 2. Since time immemorial, members of the Aition tribe have woken before sunrise to perform an elaborate ritual to their God Apotelesma. They have never missed a single day, for they operate under the (fallacious) conviction that Apotelesma is vindictive and petty, and without their daily votive, she would not allow the sun to rise. Every day, in the daylight after their morning prayer, the Aitions conclude their model of the world must be correct as it accurately predicted the sunrise.

1.1 Related Literature

Understanding the causal relationships between variables has been the focus of considerable work across many fields of study (see, for example, [Angrist and Pischke, 2009; Cunningham, 2021; Hern´an and Robins, 2020; Morgan and Winship, 2007; Parascandola and Weed, 2001; Raina K. Plowright, 2008; Pearl, 2009; Pearl, 2000; Spirtes et al., 1993], noting that this list is very cursory). Despite the considerable attention paid to both modeling and uncovering causal relationships in general, there has been much less focus on modeling subjective causal reasoning as a component of decision making. Notable exceptions include [Schenone, 2020; Ellis and Thysen, 2021; Alexander and Gilboa, 2023], who all relate subjective notions of causality to decision making, although not within the structural-equations framework.

Halpern and Piermont [2024] consider an agent who has a preference relation over interventions and provide axioms characterizing "causal sophistication," that is, when the agent's preferences can be rationalized by a utility function over variable values and a belief over structural equation models. Their representation theorem can be seen as a decision-theoretic foundation for the current framework, showing that the agent's utility function and (current) beliefs can be elicited from behavioral data.

Psychologists have studied how humans actually learn causal relationships. In simple, controlled environments, there is ample evidence that subjects' update their beliefs in a manner that approximates Bayesianism, albeit under some computational constraints [Griffiths and Tenenbaum, 2005; Kushnir and Gopnik, 2007; Bonawitz et al., 2014; Coenen et al., 2019]. It is less clear, however, how belief formation works in larger and less controlled settings, where the space of potential causal models is too large or unstructured to be considered all at once, or where the agent is simply unaware of some relevant relationships. Bramley et al. [2017] argue that people keep things simple, and "maintain only a single hypothesis about the global causal model, rather than a distribution over all possibilities and update their hypothesis by making local changes" (where, by "local changes", they mean "small changes to the causal model under consideration", such as adding more parents to a node or changing the direction of causality in one case).

Our paper lies within the more general literature on misspecified models, where an agent persistently entertains the wrong model governing data generation or the resolution of uncertainty. This topic has seen considerable recent interest within economics; see, for example, [Mailath and Samuelson, 2020; Eliaz et al., 2020; Ellis and Thysen, 2021]. A consistent theme in these papers is that agents' beliefs will often fail to converge (and in particular, fail to converge to the truth). We contribute to this literature by focusing on two novel aspects: first, we model an agent's causal judgments, that is, her beliefs about the effect of her actions. Second, we consider the possibility that the agent abandons her initial beliefs in the face of inexplicable evidence. Modelmisspecification has also been studied in strategic environments [Fudenberg and Levine, 1993; Eyster and Rabin, 2005; Esponda and Pouzo, 2016; Piermont and Zuazo-Garin, 2026]. While we do not consider strategic environments here, the extension of our framework to this setting seems straightforward.

Our approach also bears similarity to the literature on belief updating in the face of unawareness, e.g., [Karni and Vierø, 2013; Halpern and Piermont, 2020; Piermont, 2021]. In particular, we can interpret an agent who systematically neglects the causal influence of a particular variable as being unaware of the variable (or at least unaware of its relevance). For example, we could interpret the Governor in Example 1 as being unaware of the possibility that the local population might breed snakes.

2 Model

We begin by describing the general framework of structural causal models and how we model an agent's beliefs about them. At a high level, we assume a dynamic environment, where, in each period, a causal model stochastically determines the realization of all variables. The agent does not know the true causal model, but entertains probabilistic beliefs about it. Each period, the agent derives utility directly from the realized values of variables, and in addition, can take actions to intervene on variables and (partially) observe the effect of her intervention. Through this process she updates her beliefs.

The three main ingredients of our model, described in detail in the remainder of this section are: (a) a universe of variables and possible causal relations between them embodied by a set of possible structural equations; (b) a conditional probability system on this space of all structural equations that governs the agent's beliefs (and eventually her behavior); (c) the actions that are available to the agent and a notion of utility that directs the agent's choice of action.

2.1 Structural-Equations Models

We assume that the world is described by a set of variables. It is useful to split them into two sets: the exogenous variables, whose values are determined by factors outside the model, and the endogenous variables, whose values are determined by the exogenous variables and other endogenous variables. We assume that agents can intervene only on endogenous variables (but not necessarily all of them).

Let U and V denote the finite collection of exogenous and endogenous variables, respectively, that describe the environment. For each X ∈ U ∪ V, let R(X) denote the finite set of values X can take. A context is a vector ⃗u of values for all the exogenous variables U. Let R(U) = ∏Y ∈U R(Y ) consist of all contexts.

We use the standard vector notation to denote sets of variables or their values, that is, if ⃗X = (X1, . . . ,Xn) and ⃗Y = (Y1, . . . , Ym) are vectors, we write, in a standard extension of the usual containment relation, ⃗X ⊆ U ∪ V to indicate that each Xi ∈ U ∪V, or, ⃗X ⊆ ⃗Y to indicate that for each i ≤ n, there is some j ≤ m such that Xi = Yj. Similarly, we write ⃗x ∈ R( ⃗X) if ⃗x ∈ ∏i≤n R(Xi). As with the notation for random variables, uppercase letters refer to the names of variables and lowercase to their values. If ⃗X ⊆ ⃗Y and ⃗y ∈ R(⃗Y ), let ⃗y| ⃗X denote the restriction of ⃗y to ⃗X. In particular, for a single variable X, ⃗y|X denotes the Xth component of ⃗y.

A causal model is a pair M = ⟨S,F⟩, where S = (U, V,R) is a signature and F is a collection of structural equations that determine the values of endogenous variables based on the values of other variables. Formally, F = {FX}X∈V , where

Given F, we say that X depends on Y if, there is some setting of the variables in ⃗Z = (U ∪ V) - {X, Y } and values y, y′ ∈ R(Y ) such that FX(⃗z, y) ̸= FX(⃗z, y′ ); that is, changing the value of Y changes the value of X. A causal model is acyclic if there are no cyclic dependencies. Following the literature, we restrict attention in this paper to acyclic models.

We can represent qualitative features of a causal model by a causal graph, where the nodes are labeled by distinct (exogenous or endogenous) variables, and there is an edge from X to Y if Y depends on X. Note that the graph of an acyclic model is acyclic. Let Mdenote the set of all acyclic models (note that M is finite).

A setting is a pair (M, ⃗u) consisting of a model and a context. A setting (M, ⃗u) determines the value of all variables, i.e., there is a unique value for each variable that simultaneously satisfies all the structural equations in M and agrees with the context ⃗u. We write (M, ⃗u) |= ⃗Y = ⃗y for the vector ⃗y ∈ R(⃗Y ) if ⃗y is the value of ⃗Y in (M, ⃗u). See [Halpern, 2000] for details about the semantics of structural models.

We denote the intervention on variable X ∈ V that sets its value to x ∈ R(X) by X ← x, and likewise for a set of variables ⃗X ← ⃗x. This constructs a new counterfactual causal model (over the same set of variables, and consistent with the same causal graph) where the equation for each variable X ∈ ⃗X is just the constant function X = x. We denote the counterfactual model constructed by starting with M and performing intervention ⃗X ← ⃗x as M⃗X←⃗x .

2.2 Beliefs and Updating

There are two sources of uncertainty: about the true model (that is, about the causal relationship between variables) and about the context (that is, about the realizations of exogenous variables). We assume that there exists a true (but unknown) model M ⋆ that is fixed across time, so that the true causal relationships between variables remains the same at every period. By contrast, we assume that the context is stochastic, drawn i.i.d. at every period according to a probability distribution over the set of contexts, π ∈ ∆(R(U)). Therefore, the realization of the variables (without any intervention) is the outcome of (M ⋆ , ⃗u) where ⃗u is distributed according to π. The asymmetry in how we model these two sources of uncertainty captures the difference between uncertainty that arises because of persistent randomness in the system (governed by π) and the uncertainty that arises because of the agent's epistemic state (governed by the agents beliefs about the true model, as described below).

The agent may not know the true causal model M ⋆ , but we assume that the agent does know π, at least to the extent of determining those exogenous variables she is aware of. 2 Thus, if the agent learned the true model, she would learn the true distribution over variable values. She would still be unable, however, to deterministically predict the outcome, owing to the randomness of the context. 3

We now describe how we model beliefs about the true model. Let E ⊆Mbe a non-empty subset of models. Then a conditional probability system (cps) over E is a family of probability measures on the non-empty subsets of E, indexed by non-empty subsets of E; that is, µ = {µF}∅̸=F⊆E, such that for all F ′′ ⊆ F ′ ⊆ F ⊆ E, we have

The measure µF represents the conditional beliefs given evidence of the event F . The two consistency requirements state that beliefs are as dynamically coherent as possible: (CP1) states that the evidence is treated as fact while (CP2) extends a standard probabilistic identity to ensure coherence for different distributions µF .

A cps also allows beliefs conditional on events of unconditional probability 0, where the unconditional probability is given by µE. To simplify notation, we denote µE by ¯µ. The probability ¯µ completely governs the (contemporaneous) behavior of the agent, as she does not anticipate observing inexplicable evidence; the other components of the cps determine only her future behavior should her contemporaneous beliefs prove incorrect.

2 The set of causal models considered by the agent may exclude some variables, including some of the exogenous variables. Formally, we assume that the agent knows the marginal of π on those variables that are included in the models she considers possible.

3 Conceptually, it is straightforward to also allow the agent to be uncertain about the distribution π over the realization of contexts. We do not include this additional layer of uncertainty as it increases complexity and is largely unrelated to the main focus of the paper.

Shortly, we will endow the agent with the ability to take actions that consist of intervening on some set of variables and observing the effect of this intervention on (a different) set of variables. The information generated by such an action is an observation; formally an observation is a pair o = ⟨ ⃗X ← ⃗x, ⃗Y = ⃗y⟩ where ⃗X ⊆ V is the set of variables that are intervened on to be set to ⃗x ∈ R( ⃗X) and ⃗Y ⊆ U∪V is the set of variables that are observed to take values ⃗y ∈ R(⃗Y ) as a result of the intervention. We require that if Y ∈ ⃗X ∩ ⃗Y , then ⃗y|Y = ⃗x|Y . Let O denote the set of all observations.

Let the agent's beliefs be given by a cps µ. How should the agent update her beliefs subsequent to the observation o = ⟨ ⃗X ← ⃗x, ⃗Y = ⃗y⟩? In an abuse of notation, let

denote the π-probability that model M would generate the observation o. The agent can rule out any model not in the set

If the agent observes o, then she has learned the event E(o). For example, let E0 = M be the set of all models. After observing o1, the agent will learn E(o1), and entertain new beliefs modeled by a cps over E1 = E(o1)∩E0; after another observation o2, the agent will learn E(o2), and the new cps will be over E2 = E(o2) ∩ E1 = E(o2) ∩ E(o1) ∩ E0, etc.

When observing o, however, the agent learns more that just E(o): the relative likelihood of different models producing the observation yields further belief updating. For any F ⊆ E(o), let µoF be the distribution defined by

which captures the posterior probability of models using each conditional distribution as a potential prior.

We can thus define an 'updated' cps as

which can easily be checked to be a well-defined cps over E(o) whenever E(o) ∩ E ̸= ∅.

We conclude this section with a few special cases:

• The distribution π is trivial, that is, a point mass on a single context: We call such a state of affairs deterministic.
• The supports of each µF is a singleton. After any conditioning event, the agent is certain of a particular model. The results of Bramley et al. [2017] suggest that this is what people actually do.
• The cps is overMand µM is fully supported. The agent entertains all possible models and is never surprised. We could model such an agent without the cps machinery.
• The support of each µF is the set of all models defined on a causal graph g. After an observation the agent is certain about the direction of causal influence, but not the exact details of the structural equations. She only re-evaluates her world view when she cannot explain the data with her given graph g.

2.3 Agency and Utility

Each period, the agent can take an action; formally, an action is an intervention paired with a set of variables to observe: a = ( ⃗X ← ⃗x, ⃗Y ). The interpretation is that by taking the action a the agent intervenes so as to set the variables ⃗X ⊆ V to the values ⃗x ∈ R( ⃗X). In addition, the action a allows the agent to (partially) observe the effect of the intervention; she observes the value ⃗y ∈ R(⃗Y ) that results from the intervention. Let A denote the set of all actions available to the agent.

Given a setting (M, ⃗u), each action a = ( ⃗X ← ⃗x, ⃗Y ) determines the observation that would be made: if a is taken, then given (M, ⃗u), the agent will expect to observe

where ⃗y ∈ R(⃗Y ) is the unique vector of values such that (M⃗X←⃗x , ⃗u) |= ⃗Y = ⃗y. Let µaM,⃗u := µoaM,⃗u denote the cps that results from updating on this observation.

Finally, let UTIL ∈ V denote a distinguished variable which we interpret as utility. In an abuse of notation, we can treat UTIL as a utility function over outcomes of variables as given by settings. This is defined by UTIL(M, ⃗u) = x for the x ∈ R(UTIL) such that (M, ⃗u) |= UTIL = x. While it is not required by what follows, it is perhaps reasonable to assume that for all ( ⃗X ← ⃗x, ⃗Y ) ∈ A, we have UTIL ∈ ⃗Y ; that is, independent of whatever else is observed, the agent can determine her own utility.

For an action a = ( ⃗X ← ⃗x, ⃗Y ) ∈ A, let M a = M⃗X←⃗x , be the counterfactual model induced by the action a. With this notation, we can further extend UTIL to capture the agent's (one-shot) value of taking action a, given her beliefs µ:

The decision maker, however, does not only care about her one-shot value but also her ability to continue to make choices. Given a discount factor of δ ∈ (0, 1), the agent values the action a according to its one-shot utility and its information value. This leads to our recursive definition of VAL, as is standard in infinite horizon dynamic programming.

This is well defined on the basis that UTIL is bounded and δ < 1 (see, e.g., Theorem 4.6 of [Stokey and Lucas Jr., 1989]). The value of taking action a ∈ A today is the expectation of

• UTIL(a, µ) - the immediate payoff associated with the intervention according to today's utility, and
• δmaxa′∈A VAL(a′ , µaM,⃗u ), the discounted value of the same problem tomorrow, except that beliefs will be updated according to the additional observation made. Recall that oaM,⃗u is the observation that is realized when action a is taken given model M and context ⃗u, so µaM,⃗u is the agent's updated beliefs conditional on what she observes as part of her chosen action.

Note that we are assuming, by nature of the "max" operator in the continuation value (i.e., tomorrow's utility), that the agent optimally uses whatever information arrives to maximize the continuation value. Note, the actual observation made by the agent is determined by M ⋆ . So, if M ⋆ is not in the support of ¯µ, it is possible that the true observation oaM⋆ ,⃗u does not coincide with any oaM,⃗u considered with positive probability; that is, ¯µ(E(oaM⋆ ,⃗u )) = 0. This is not considered possible by the agent (i.e., it is a 0-probability event, and therefore does not contribute to the choice of a ∈ A), but is possible from the modeler's perspective. However, the agent can still condition on the event using her cps.

3 Steady State

We now introduce the idea of a steady state, where the optimal action today does not alter beliefs, so that it remains optimal tomorrow. Therefore no new information enters the system and actions are repeated. Using the notation of the previous section, the environment (given a set of variables and ranges) is described by the tuple

Given this, a steady state is a pair (µ, a) consisting of a cps µ, and an action a such that:

• a ∈ argmaxa′∈A VAL(a′ , µ);
• ¯µ = ¯µaM⋆ ,⃗u for ⃗u ∈ R(U) with π(⃗u) > 0.

The first condition states that the agent is choosing an optimal action given her beliefs. The second condition states that, given this action and the true model, the realized observation does not change her unconditional beliefs-hence the same action will remain optimal in the future.

The character of a steady state is quite straightforward in deterministic environments (i.e., when π is trivial), as shown by the following two remarks, neither of which holds in the more general stochastic environment, as we show below.

Remark 1. Suppose that π is trivial. If (µ, a) is a steady state, then a ∈ argmaxa′∈A UTIL(a′ , µ).

Proof. Let ⃗u be the unique context in the support of π. Then, since observing oaM⋆ ,⃗u does not change her beliefs, it must be that oaM,⃗u = oaM⋆ ,⃗u for every model M in the support of ¯µ. As such, with ¯µ × π-probability 1, VAL( ·, µaM⋆ ,⃗u ) = VAL( ·, µ). The remark follows immediately.

This remark does not hold if the context is stochastic. In particular, it is possible that an agent would choose a statically suboptimal action because she thinks there is some probability it will be informative (i.e., there are some contexts in which it would allow her to discriminate between models), while in fact (i.e., according to M ⋆ ) it is not informative for any context.

Our second remark shows that, in a deterministic environment, the agent's dynamic behavior will converge to a steady state.

Remark 2. Assume that π is trivial and let ⃗u be the unique context in the support of π. Let µ be any cps over E for some E containing the true model M ⋆ . Starting with µ0 = µ, define inductively the following sequence of actions and beliefs: for each i ≥ 1, take 4

• ai ∈ argmaxa′∈A VAL(a′ , µi-1);
• µi = (µi-1)aiM⋆ ,⃗u .

Then (µi, ai) converges to a steady state as i →∞.

Proof. Given the triviality of π, ¯µi = ¯µi-1 if and only if oaiM,⃗u = oaiM⋆ ,⃗u for every model in the support of ¯µi-1; as such, any belief change corresponds to the agent excluding some previously considered model. Since there is a finite number of models, it follows that beliefs are eventually constant; let k > 0 be such that for all i ≥ k, ¯µi = ¯µk. Then by construction we have

• ak+1 ∈ argmaxa′∈A VAL(a′ , µk) and
• ¯µk = ¯µk+1 = (¯µk)ak+1 M⋆ ,⃗u ,

and so, a steady state.

As mentioned above, Remark 2 does not hold in general, when there is uncertainty about the context. In fact, not only might repeated maximization not lead to a steady state, it does not necessarily even lead to stable behavior. In other words, it is possible that the agent's optimal action cycles between alternatives forever. We provide a simple example of such an environment in Example 3.

4 Examples

To get a sense of what a steady state entails, we first formalize the examples from the introduction within this framework. We also provide an example of the failure of convergence for non-deterministic environments. As it is relatively simpler, we begin with Example 2. 5

Example 2. (continued) There are three endogenous variables, V = {UTIL, PRAY, SUN} and no exogenous variables U = ∅. Both (non-utility) variables have range {0, 1}. The first, PRAY, represents whether the Aitions make their daily votive (PRAY = 1 if they do) and the second, SUN, whether the sun rises (SUN = 1 if it does). Consider the following two structural models (which differ only in the equation that governs SUN):M ⋆ : PRAY = 0, SUN = 1,

and

The true model is M ⋆ . The Aitions, however, consider M † . Let µ be a cps such that ¯µ(M † ) = 1 and, for any E with M ⋆ ∈ E and M † /∈ E, µE(M ⋆ ) = 1.

4 We must assume ties are broken consistently: if VAL( · , µj) = VAL( · , µj′ ) then aj+1 = aj′+1.

5 In these examples, for ease of exposition, we omit the exogenous variables, which determine the relevant endogenous variables in the obvious way.

The Aitions have two actions at their disposal:

The associated model-dependent predicted observations are: 6

The observation generated by action a1 does not depend on the model, whereas the observation generated by a0 is model dependent. Thus, starting from an initial belief ¯µ, the action a1 generates an observation that does not update beliefs at all; that is ¯µa1 M⋆ = ¯µ. On the other hand, observing o0m⋆ = ⟨PRAY ← 0, (PRAY = 0, SUN = 1)⟩ cannot be explained by ¯µ, leading to wholesale belief revision to µE(o0m⋆), and so, given the structure of µ, adoption of correct beliefs.

Finally, given ¯µ, the immediate value of each action is given by UTIL(a, µ) = UTIL(M †,a ), so that UTIL(a1, µ) = 9, whereas UTIL(a0, µ) = 0. So, (µ, a1) is a steady state: given their initial beliefs, the Aitions will continue to pray every morning, and consequently, never update their beliefs. 7

In a similar manner, we can formalize Example 1:

Example 1. (continued) There are five variables, V = {UTIL, SNAKE, BOUNTY, BREED, HUNT}, where SNAKE represents the snake population (with range {0, 1, 2}), BOUNTY represents the existence of a bounty (with range {0, 1}), HUNT represents whether locals hunt snakes (with range {0, 1}), and BREED represents whether locals breed snakes (with range {0, 1}).

Consider the following two structural models.

Again, the true model is M ⋆ , but the Governor considers M † . InM † , the variable BREED is considered irrelevant (i.e., the agent is unaware of or inattentive to that variable). As in Example 2, let µ be a cps such that ¯µ(M † ) = 1, and, for any E with M ⋆ ∈ E and M † /∈ E, µE(M ⋆ ) = 1. The Governor must choose between

6 Since there are no exogenous variables, we suppress the subscript for the context. Further, we abuse notation and write oi rather than oai for the action superscript.

7 Note also that they place probability 0 on ever receiving any information, so the choice of action is determined entirely by the immediate value: VAL(a1, µ) = 9 1-δ and VAL(a0, µ) = 0

with the associated predicted observations

As in Example 2, the observation generated by one action (here a0) is model independent, whereas the observation generated by the other action (here a1) depends on the model. In contrast to Example 2, however, the action chosen under the initial beliefs is the revealing action. To see this, note that, since the Governor places probability 1 on M † , he sees no value in exploration, and thus chooses his action based only on the immediate payoff. Since UTIL(a1, µ) = -1 and UTIL(a0, µ) = -2, the Governor institutes the bounty, expecting the snake population to decline. However, the actual observation is o = o1m⋆: an increase in the snake population (and a realized utility of -5).

This observation cannot be explained by his current model, so the Governor would revise his beliefs to µE(o). Under his new beliefs, his optimal action is to retract the bounty, which corresponds to steady-state behavior.

Notice that in this example, when the value of BREED is fixed to 0, the equations of M ⋆ reduce to the equations of M † . That is, one could view M † as a specification of M ⋆ where a particular variable is ignored (i.e., is implicitly deemed unchangeable, is believed to bear no causal relation to any utility-relevant variable, or lies outside of the agent's awareness). Interpreting M † as capturing the beliefs of an agent who is unaware of BREED and M ⋆ as encoding her understanding after becoming more aware, then this equational relation between the two models embodies a principle akin to the principle of Reverse Bayesianism [Karni and Vierø, 2013] in state-space models of awareness. That is, when awareness expands, the previously-held relationships between variables remain as a special case of the new expanded relationships.

Example 3. Let U = {Y } be a single variable with range R(Y ) = {0, 1, . . . , 99}, and π the uniform distribution over outcomes. There are two endogenous variables, V = {UTIL,X}, where X is the only variable that can be intervened on, and UTIL is the agent's utility with ranges R(X) = {r, l,m} and R(UTIL) = {0, 1}. We consider three causal models:

Suppose that there are two actions:

Note the following simple observations: first, for beliefs sufficiently concentrated on M r (resp., M l ), the action ar (resp., al) maximizes expected utility; second, since the agent cannot observe Y , there are no inexplicable outcomes; all three models can produce any combination of X and UTIL. As such, she will never completely discard any model that is in the support of her initial beliefs.

Finally, assume the support of the agent's initial beliefs, ¯µ, is {mr ,ml }. In this case, her behavior will switch between the two actions infinitely often. To see this, assume to the contrary that her behavior was eventually constant. Without loss of generality, assume she eventually chooses ar forever. Under the true model M ⋆ , this produces the outcome (X ← r, UTIL = 1) with probability 1 2 and the outcome (X ← r, UTIL = 0) with probability 12 . This is the expected distribution of observations under M l , but not under M r ; thus, with π-probability 1, with sufficiently many observations, her beliefs will become arbitrarily concentrated on M l ; however, for such beliefs, ar is no longer optimal.

5 Introspective Unawareness

The agents in the framework laid out above entertain a classical exploration/exploitation trade-off: given their initial uncertainty about the true causal model, the forward looking agents maximizing VAL( ·, µ) can take statically suboptimal actions to gather information about the causal structure, enabling better future decisions. The perceived value of exploration is determined by the agent's current beliefs, ¯µ, and so she acts with certainty that the true model is contained in the support of ¯µ. This is true even if the agent has repeatedly abandoned prior beliefs in the face of surprising observations; she never considers the possibility that her current beliefs might be misspecified or incomplete.

In this section, we augment the model above slightly to accommodate an introspectively unaware agent, who entertains the possibility that her beliefs are incomplete. If the agent was aware of which relevant models she was failing to consider, she could simply shift some small probability mass onto these models, expanding the support of her beliefs to capture all relevant possibilities. By way of contrast, we assume that the agent is (a) unaware of the models outside the support of ¯µ, but (b) aware that she is unaware. This parallels awareness of unawareness in state-space models [Karni and Vierø, 2017]. The agent therefore cannot predict what it is that she might observe that would make her abandon her current world view. Nonetheless, the relevant parameters may still be identified from choice behavior [Karni and Vierø, 2025].

In essence, the idea is that the agent considers the possibility of encountering some inexplicable evidence, and conjectures about the potential likelihood and value of doing so, but cannot explicitly determine what the world would look like afterwards. To model this, we introduce two new objects into our model.

• Let v∗ denote the agent's subjective value of learning something unforeseeable, capturing the agent's best guess of the (discounted expected) value of a paradigm shift that causes her to abandon her current beliefs.
• Let ξ : A → [0, 1] denote a function that represents the agent's beliefs about the likelihood of each action yielding a surprising outcome, and let τ : [0, 1]A ×O → [0, 1]A describe how the agent updates her beliefs about the likelihood of surprise.

With these two pieces of notation, we can consider the introspective agent's version of VAL:

The value of an action a is the ξ(a) convex combination of v∗-the value subjectively assigned to unknown outcomesand the expected value of the problem under the distribution ¯µ. This second component is almost the same as VAL as defined in Section 2.3, except that the continuation value also allows for the possibility of discovering a novel outcome, where the probabilities of discovery have been updated by τ .

Notice that the appeal of the novel outcome only kicks in when the known part of the model-space provides low expected payoff. In line with the finding of Bonawitz et al. [2014], actions that yield low "objective" payoff but high chance of revealing something novel will only be desirable when actions with higher objective payoff are unavailable.

We can therefore recast our notion of a steady state using v∗ : Call a tuple (µ, a, ξ) an introspective steady state if

• a ∈ argmaxa′∈A VAL∗ (a′ , µ, ξ)
• ¯µ = ¯µaM⋆ ,⃗u for ⃗u ∈ R(U) with π(⃗u) > 0, and
• τ(ξ, oam⋆ ,⃗u ) = ξ for ⃗u ∈ R(U) with π(⃗u) > 0.

The first two conditions are direct analogs of the conditions of a (non-introspective) steady state. The third also requires that the probabilities of discovery remain unchanged.

6 Conclusion

In Sections 2 and 3, there are no behavioral markers of offpath behavior; choices are dictated entirely by ¯µ (and its onpath evolution), as the decision maker places 0 probability on entertaining incorrect beliefs. Nonetheless, we do believe there are normative criteria for choosing one cps over another. For example, the agent may may prioritize structural stability (e.g., retaining the same causal graph when possible), or may want to move to beliefs that are 'close' to her old beliefs (e.g., changing as few structural equations as possible). We hope to formalize these ideas in future work.

Another direction we are pursuing is to characterize the exact boundary demarcating convergence to a steady-state. A main obstacle for convergence, as underscored by Example 3, is when an action a is optimal conditional on model M, but under the true model M ⋆ taking a results in observations that make M less likely. Eliminating this yields convergence, but this is not necessary. Another interesting question regards convergence in distribution: the agent's behavior may not converge to a constant action but rather to a distribution over actions.

Acknowledgments

We would like to thank seminar audiences at the Philosophy, Politics, and Economics Society London Meeting and at Time, Uncertainties, and Strategies XI. Vierø gratefully acknowledges financial support from Aarhus University Research Foundation.