A Ladder of Causal Distances

1 Introduction

Reasoning about the causes and effects driving physical and societal phenomena is an important goal of science. Causal reasoning facilitates the prediction of intervention outcomes and can ultimately lead to more principled policymaking [Spirtes et al., 2000; Pearl and Mackenzie, 2018].

Given a causal model, reasoning about cause and effect corresponds to formulating causal queries, which have been organized by Pearl and Mackenzie [2018] in a three-level hierarchy termed the "ladder of causation": observational queries correspond to seeing and observing; interventional queries correspond to acting and intervening; and counterfactual queries correspond to imagining, reasoning, and understanding.

Asking such questions requires a causal model to begin with. Inferring a causal model from observational, interventional, or mixed data is the problem called causal discovery.

Much of science is concerned with causal discovery, and automating the task has been receiving increased attention in the machine learning community [Peters et al., 2017], where causal models have become the tool of choice for tackling important problems such as transfer learning, generalization beyond spurious correlations [Rojas-Carulla et al., 2018], and algorithmic fairness [Kusner et al., 2017].

Evaluating causal discovery algorithms requires comparing the inferred causal models to ground-truth models on benchmark datasets, which in turn requires a notion of distance between causal models. When defining such a distance, it is not sufficient to rely on tools developed for comparing standard generative models, such as goodness of fit, as these tools only operate on the first, observational level of the ladder of causation. Remarkably little research has been done on the topic of evaluating and comparing arbitrary causal models higher up the ladder. Existing works focus on specific aspects, such as the outcome of a limited number of interventions manually selected in advance [Singh et al., 2017] or the graph structure of the models being compared [Peters and Bühlmann, 2015]. The latter work, which proposed the structural intervention distance (SID), one of the most prominent causal distance measures, further assumes that the two models have an identical observational joint distribution. Unfortunately this assumption rarely holds in practice, and we show that even if the observational joint distributions are just slightly different, SID cannot be trusted. Furthermore, previous causal distances do not cover the counterfactual level.

To close this gap, we introduce three distances (Sec. 4), one for each rung of the ladder of causation. Our distances measure the difference between causal models for each type of causal query (observational, interventional, counterfactual). Each distance builds upon the distance one level below, thus mirroring the hierarchy of the ladder of causation. We highlight theoretical properties of the distances in relation to previously proposed distances (Sec. 5). Then, we study their behavior in a series of experiments and put them to use in evaluating existing causal discovery systems (Sec. 6). Code for reproducing our experiments 1 and an extended version of the paper (with added appendices) 2 are available online.

1 https://github.com/epfl-dlab/ causal-distances

2 https://arxiv.org/abs/2005.02480

2 Preliminaries

2.1 Causal Graphs

We consider a finite ordered set of random variables X = {X1, . . . ,Xd}. A directed graph G = (X,A) consists of the set of indexed nodes X together with a set of directed edges A⊆X×X. If (Xi,Xj) ∈ A, we say that Xi is a parent of Xj and denote the set of all parents of Xj with PAj. If G contains no directed cycle it is called a directed acyclic graph (DAG).

DAGs are often used to encode causal assumptions by viewing an edge (Xi,Xj) as the statement "Xi is a direct cause of Xj" [Pearl, 2009]. A graph associated with such causal interpretation is called a causal graph.

2.2 Structural Causal Models (SCMs)

A structural causal model C is a tuple (X,N,F,PN), where PN is a noise distribution over the (exogeneous) noise variables N and F = { f1, . . . , fd} is an ordered set of structural equations indicating, for each Xi ∈ X, how its value is determined by its parents and noise:

where PAi ⊆X, and Ni is the noise variable associated with Xi. The noise models variations due to ignored variables or inherent randomness. We assume that the noise variables are independent [Pearl, 2009]. The associated causal graph G is obtained by viewing each variable in X as a vertex and drawing an arrow from each parent in PAi to Xi.

Assumptions. Throughout the paper, we assume that all models satisfy four standard assumptions: the Markov property, causal minimality, causal faithfulness, and positiveness [Peters et al., 2017].

2.3 Observational, Interventional, and Counterfactual Distributions

Fig. 1 is a graphical illustration of a three-variable SCMwith queries about the observational, interventional, and counterfactual distributions. We define these distributions next.

Observational. A causal model C entails a unique joint distribution of X = (X1, . . . , Xd) called the observational distribution and noted P C X [Peters et al., 2017]. To sample from C, we can simply sample from the noise distribution PN and use the structural assignments (cf. Eq. 1) following the topological order of X in the causal graph G .

Interventional. An intervention on the set of variables I ⊂ X of the causal model C consists of replacing the structural assignments (cf. Eq. 1) of variables in I by forcing them to specific values I = i, a so-called hard intervention. 3 The new causal model obtained from C via the intervention I = i is denoted by C;do(I = i) [Pearl, 2009]. Graphically, the interventional model is obtained by removing all incoming edges to the nodes in I. After sampling the noise and following the new structural assignments, we obtain samples from the interventional distribution of X, denoted by P C;do(I=i) X .

3 For simplicity, we focus on hard intervention. However, the approach can easily be extended to soft interventions.

Counterfactual. At the counterfactual level, we first (partially) observe the causal model in some state E = e, where E⊆X is called the evidence set. Then we ask: "Given that E = e actually happened, what would have happened had we done the intervention I = i?" This is different from the interventional level, where we only ask: "In general, what happens if we do I = i?" We now take into account the additional specific information provided by the evidence E = e.

Consider the causal model C with noise distribution PN for which we have some evidence E = e. The counterfactual model induced by C and E = e is denoted by C|E = e and is identical to C except for the noise distribution PN|E=e which has been updated given the evidence using Bayes' rule [Pearl, 2009]:

The updated noise variables are not necessarily independent anymore. Note the difference in notation between the induced counterfactual model C|E = e and the induced interventional model C;do(I = i). The former corresponds to updating the noise distribution, whereas the latter corresponds to modifying the structural assignments of variables I.

A counterfactual query corresponds to an intervention do(I = i) in the counterfactual model C|E = e. Again, this intervention entails a distribution of X, called the counterfactual distribution and denoted by P C|E=e;do(I=i) X .

2.4 Metrics, Pseudometrics, and Premetrics

A metric d satisfies the four axioms of non-negativity (d(x,y) ≥0), identity of indiscernibles (x = y ⇐⇒ d(x,y) = 0), symmetry (d(x,y) = d(y,x)), and the triangle inequality (d(x,z) ≤ d(x,y)+d(y,z)). The causal distances introduced in this paper are pseudometrics (i.e., they relax the identity of indiscernibles), whereas SID (Sec. 1 and 3) is a premetric (i.e., only non-negativity and x = y =⇒ d(x, y) = 0 hold).

3 RelatedWork

An important practical application of causal-model distances is the evaluation of causal discovery techniques. Ideally, one would like to compare an inferred causal model against a given ground-truth model for each type of causal query: observational, interventional, and counterfactual.

The comparison of observational distributions has been studied extensively in machine learning and statistics (cf. the overviews by Theis et al. [2015] and Sriperumbudur et al. [2010]), and distances between distributions have been used to evaluate causal discovery methods, typically by measuring the goodness of fit of the observational distribution induced by a model with respect to empirical samples from the true observational distribution (cf. Singh et al. [2017] for an overview). Importantly, such methods are inherently limited to the observational level and cannot measure how well the inferred causal model performs at the interventional and counterfactual levels.

These two levels have received relatively little attention, compared to the observational one. We are not aware of any previously proposed causal-model distance to consider the counterfactual level, and the few that consider the interventional level focus on a specific aspect of causal models: their causal graphs [de Jongh and Druzdzel, 2009]. For instance, the popular structural Hamming distance (SHD) [Acid and de Campos, 2003] counts in how many edges the two input graphs differ. Peters and Bühlmann [2015] argue that previous graph comparison metrics, including SHD, are not in line with the end goal of causal discovery, namely, predicting the outcome of interventions, and propose the structural intervention distance (SID), a premetric (cf. Sec. 2.4) that counts the number of pairwise interventional distributions on which two causal models (with graphs G and H , respectively) disagree:

Under the assumption that the causal models agree on an underlying observational distribution, Peters and Bühlmann [2015] show that the comparison of interventional distributions reduces to a purely graphical criterion. In particular, when the graphs are the same, both SHD and SID are 0, and SHD(G ,H ) = 0 implies SID(G ,H ) = 0.

Acharya et al. [2018] compare two causal models, but only for the purpose of testing identity. This constitutes a special case of our interventional distance. Recently, Gentzel et al. [2019] argued that causal discovery methods should be evaluated using interventional measures instead of structural ones such as SID and SHD. The causal distances we introduce are interventional and counterfactual measures.

3.1 Limitations of RelatedWork

Even though SID is focused on interventional distributions, it assumes that the underlying observational distribution has been estimated correctly. In practice, this is usually not the case, since the estimation is done using finitely many noisy samples. In general, SID cannot provide useful answers when the causal models disagree at the observational level. In fact, even when the observational distribution is just slightly off, SID may still produce highly inaccurate results.

To illustrate this problem, consider two causal models C1,C2, each with two nodes A,B. Both models have the graph A→B, with A ∼N (0,σA) and B's noise NB ∼N (0,σB):

The two models predict different values for the intervention do(A=a). In a toy interpretation, B could be the improvement in life expectancy, and A the daily intake of some drug. Then these two models would give rise to opposite policies given the goal of maximizing life expectancy. This should be reflected by a large distance between the models, but in fact the opposite happens: since C1 and C2 share the causal graph G , we have SHD(G , G ) = SID(G , G ) = 0.

Strictly speaking, SID cannot even be applied in this case because the observational distributions are not identical. If, however, σA ≪ σB, the observational distributions become almost indistinguishable, and one might be tempted to apply SID, obtaining a distance of 0 although the interventional distributions still give rise to opposite policies. We provide more details about this problem and its resolution in Appendix A (cf. footnote 2).

Another limitation of SID is that it is binary: either two pairwise interventional distributions are the same or not (cf. Eq. 4). It does not quantify the difference. In fact, for practical applications, two slightly wrongly inferred interventional distributions might be preferable to one completely wrongly inferred distribution. Also, if one has prior knowledge about which interventions are more critical, one might want to reflect this in the evaluation. Finally, SID and SHD cannot compare causal models at the counterfactual level as they ignore the structural equations and noise distributions (cf. Eq. 1).

4 Definition of Causal Distances

Let X be a set of random variables, and C1,C2 two causal models defined over them. We now introduce natural formulations of distances at the observational, interventional, and counterfactual level. Intuitively, they build upon an underlying distance between probability distributions and mirror the hierarchical aspect of Pearl and Mackenzie's ladder [2018].

4.1 Observational Distance (OD)

Let P C1 X ,P C2 X be the observational distributions induced by C1,C2. The observational distance (OD) is trivial and corresponds to choosing a distance D between probability distributions:

Example choices for D include the Hellinger, total variation, or Wasserstein distance.

4.2 Interventional Distance (ID)

An intuitive way to compare two causal models C1,C2 at the interventional level is to compare all their interventional distributions. Let I denote the node on which the intervention is performed and µ a distribution over nodes that weighs the interventions on each node. In the absence of such information, µ may be chosen as the uniform distribution. Then, the interventional distance (ID) is defined as

By convention, we include the empty intervention I = /0, which corresponds to the observational distance OD.

In words, ID is the expected deviation in the interventional distributions if we sample a node I on which to intervene according to µ and sample its value according to PI .

The expectation Ei∼PI indicates that I's values are drawn from the distribution PI . For instance, PI can be uniform for discrete models or standard Gaussian for continuous models. We only enforce PI to be strictly positive for all possible values that I can take.

Variables in X may have different value scales, such that the averaged distances may be dominated by a few variables. This can be fixed by co-opting the weights µ in order to normalize each variable. In the tool we release, we give the option to z-normalize each variable and compute a scale-invariant ID.

Note that µ can give weights of 0 to some nodes, e.g., if they are unobservable to the causal discovery method.

4.3 Counterfactual Distance (CD)

The natural way to compare models at the counterfactual level is to consider their interventional distance on all counterfactual models, i.e., the counterfactual models induced by all possible evidences. Let E = e denote the observed evidence and ν a distribution over nodes that weighs the counterfactuals induced by observing each node. Similar to µ, in the absence of further information, ν may be chosen to be uniform. Then, the counterfactual distance (CD) is defined as

By convention, we include the empty evidence E = /0, which corresponds to the interventional distance ID.

The expectation Ee∼PE indicates that E's values are drawn from the distribution PE. In the absence of information, PE may be uniform for discrete models or standard Gaussian for continuous models.

5 Basic Properties of Causal Distances

In this section, we assume that both µ and ν are uniform over the set of nodes. The proofs are given in Appendix F.

Each distance builds on top of the distance defined at the level below (CD on ID; ID on OD), thus reflecting the hierarchical structure of the ladder of causation. Furthermore, one can verify the following connections.

Theorem 1. For two causal models C1 and C2 over the variables X, we have:

In particular, counterfactual equivalence implies interventional equivalence, which in turn implies observational equivalence.

5.1 Connection with Graph-basedMetrics

The interventional distance (ID) is related to the graph-based SID and SHD via

Theorem 2. For two causal models C1,C2 with causal graphs G1, G2,

The reverse directions do not hold in general.

A further connection between SID and our causal distances is given by

Theorem 3. Let C1,C2 be two causal models with causal graphs G1, G2. When OD(C1, C2) = 0,

When OD(C1,C2) ̸= 0 the equivalence does not hold.

From Thm. 2, we know that ID being 0 guarantees that SID is 0. But SID being 0 only ensures that ID is 0 in the specific case where OD is also 0.

5.2 Hidden Variables

Until now, we considered the comparison of two Markovian causal models, i.e., with no hidden confounders. We may ask what happens in the non-Markovian case, where one or both models have hidden confounders.

If both models have hidden confounders that can be intervened on, we cannot bound the expected difference between two models, as the outcome of intervening on the hidden confounder can be made arbitrarily large, as shown by Fig. 2.

However, this constitutes a fairly peculiar scenario. Indeed, it is expected that comparing "incomplete" models can only give partial information. It is similar to trying to establish a distance between vectors where one dimension remains hidden. In practice, if we wish to compare two causal models either (i) one is fully known (e.g., the gold standard model to which we compare a model inferred by a causal discovery technique with variables unobservable during training) or (ii) the hidden variables cannot be intervened on. In this latter case, OD and ID computed on the observed subset preserve their interpretation.

5.3 Practical Implementation

In Appendix D, we describe the practical details related to the implementation and efficient estimation of the causal distances. We also discuss how to handle both continuous and discrete variables. This implementation results in a tool that we make publicly available to the community.

6 Experiments

We now conduct experiments with the causal distances, on both synthetic and real-world causal models. In all experiments, we use the sample Wasserstein distance [Villani, 2008] as the underlying distance D between probability distributions (cf. Eq. 7). In Appendix B, we validate the sample efficiency and sensitivity of the causal distances.

6.1 Geometry of Causal Distances

First, we illustrate the intuitive geometry induced by our causal distances. In particular, we consider simple models with two nodes A and B using linear structural equations and Gaussian noise. We let β > 0 denote the strength of the causal connection, let N ∼N (0,1) be B's noise, and consider four types of models:

We construct five models of each type with β = 0.1, 0.5, 1, 2, 5, respectively, resulting in 20 models overall. We then compute the pairwise distances between all models using ID and apply multidimensional scaling to obtain 2D embeddings of all models. The result is depicted in Fig. 3a and exhibits the geometrical structure induced by ID, where each type of model creates its own branch, and larger values of β push the different types further apart. When β →0, all models converge to a model where A and B are causally disconnected. When viewed in 3D, equal angles would separate all pairs of branches.

In contrast, SID induces a much poorer geometry where each model is projected on one of two points: one representing the graph A→B, the other, the graph B→A. With OD, shown in Fig. 3b, the models form one branch in the 2D embedding. They are only distinguished based on the amplitude of β; neither the sign nor the orientation of the graph are captured.

6.2 Comparison of Causal Distances and SID

Whereas Thm. 3 connects ID, OD, and SID when two of these quantities are 0, we now empirically investigate their relationship when they deviate from 0. Fig. 4a shows a scatter plot comparing ID and SID, where each dot is a pair of random causal models. As we see, there is little correlation between ID and SID. It is possible to find pairs of causal models with low ID but high SID, and vice versa. These results highlight how the different distances capture different aspects of the models being compared.

6.3 Evaluation of Causal Discovery Systems

An important application of causal distances is the evaluation of causal discovery systems. In this section, we illustrate this by evaluating several causal discovery systems using both real-world and synthetic causal models.

We considered the following real-world Bayesian causal models from Elidan [2001]: Cancer1, Cancer2, Earthquake, Survey, Protein, Child, and Insurance. They range from 5 to 27 nodes and from 4 to 52 edges. For synthetic models, we sample random DAGs and random parametrizations using the CDT tool [Kalainathan and Goudet, 2019]. We consider the following parametrizations: linear Gaussian (linGauss), linear non-Gaussian (linNGauss), Gaussian process with additive noise (GPAddit), and Gaussian Process (GP).

For each model, we sample 2,000 observations from which causal discovery methods should recover the causal model. Since they are causal Bayesian networks, not structural causal models, we cannot compute counterfactuals [Pearl, 2019]. Thus, we restrict ourselves to ID, SHD, and SID.

Systems. We consider multiple causal discovery methods for recovering the causal graph: CCDr [Aragam and Zhou, 2015], PC [Spirtes et al., 2000], GES [Chickering and Meek, 2002], GIES [Chickering, 2002], MMPC [Tsamardinos et al., 2003a], IAMB [Tsamardinos et al., 2003b], LiNGAM [Shimizu et al., 2006], and CAM [Spirtes et al., 2000]. Some of these techniques only output a partial DAG with undirected edges, and some only output a graph without parameters. To obtain a fair comparison of the full Bayesian networks, we perform all parameter estimation via maximum likelihood estimates based on the training data. When only a partial DAG is returned, we use the edge orientation that provides the best goodness of fit after the parameters have been estimated. Alternatively, one could report the mean performance over all possible directed graphs.

Remark. Our distances compare full causal models, i.e., causal graphs after the parameters have been estimated. Here, by fixing the parameter estimation, we measure the impact of the causal graph on the intervention predictions. It also ensures that two methods that output the same graph (DAG or partial DAG) will obtain the same evaluation results. Yet, as shown below, it does not mean that these evaluation results agree with graph-based metrics such as SHD or SID. For the systems, we use the implementations available in CDT. We compute maximum likelihood estimates using the Pomegranate python framework.

Results. In Fig. 5, we report the evaluation results broken down by model parametrization. The Real Networks block corresponds to the performance of systems averaged over the real-world causal models described above. Results for each causal model are available in Appendix C. We see that different metrics yield different rankings of systems. Thus, the differences between metrics observed in Sec. 6.2 are also consequential in the task of causal discovery. These observations emphasize the importance of employing the evaluation metric that captures the desired behavior. If only the observational distribution matters, OD should be used, but then causal discovery may not be needed in the first place. If we care about the expected errors in predicting the outcome of interventions, ID should be used, and SID can be employed when we focus on the causal graph under the assumption that the underlying observational distribution has already been correctly estimated. Additionally, even a single metric produces different rankings of systems in different scenarios. Causal discovery requires assumptions about the underlying structure of the true causal model, and few guarantees are given when the respective assumptions are not met. Different networks fulfill different assumptions and are best handled by different causal discovery methods. An evaluation using causal distances such as OD, ID, and CD is indispensable for illuminating which causal discovery method is best suited for which kind of data. Interestingly, ID clearly reveals that systems struggle most for the linear Gaussian case, which is known to be unidentifiable. While all other cases are identifiable, Gaussian processes with additive noise (GPAddit) seem to be the easiest for existing causal discovery systems. Overall, CCDr seems to perform fairly well in comparison to other systems. In Appendix E, we show that, contrary to machine learning, causal discovery systems do not benefit from more training data.

7 Conclusion

This paper introduces observational (OD), interventional (ID), and counterfactual (CD) distances between causal models, one for each rung of the ladder of causation (cf. Sec. 1). Each distance is defined based on the lower-level ones, reflecting the hierarchical structure of the ladder. We study the properties of our distances and propose practical approximations that are useful for evaluating causal discovery techniques. We release a Python implementation of our causal distances.

Our causal distances do not require the unrealistic assumptions of infinite samples and perfect statistical estimation that are currently common in the study of causality [Pearl, 2009]. Also, they quantify the difference between causal models on a continuous, rather than integer, scale and make use of the data at a finer granularity than the usual binary measurements of methods such as SHD and SID (cf. Sec. 3).

The proposed causal distances have both theoretical and empirical applications, and we hope the research community will use them to advance the study of causality.

Acknowledgments

With support from Swiss National Science Foundation (grant 200021_185043), European Union (TAILOR, grant 952215), and gifts from Google, Facebook, Microsoft