--- title: "Causes of Effects: Learning Individual Responses from Population Data" authors: - "Scott Mueller" - "Ang Li" - "Judea Pearl" doi: "10.24963/ijcai.2022/376" url: "https://ijcai.org/proceedings/2022/376" pdf: "https://ijcai.org/proceedings/2022/0376.pdf" venue: "Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence" year: 2022 pages: "2712-2718" track: "Main Track" keywords: - "Uncertainty in AI: Causality, Structural Causal Models and Causal Inference" - "Knowledge Representation and Reasoning: Causality" - "Uncertainty in AI: Graphical Models" --- # Causes of Effects: Learning Individual Responses from Population Data Scott Mueller, Ang Li, Judea Pearl *Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence*, pages 2712-2718. DOI: [10.24963/ijcai.2022/376](https://doi.org/10.24963/ijcai.2022/376) ## Abstract The problem of individualization is crucial in almost every field of science. Identifying causes of specific observed events is likewise essential for accurate decision making as well as explanation. However, such tasks invoke counterfactual relationships, and are therefore indeterminable from population data. For example, the probability of benefiting from a treatment concerns an individual having a favorable outcome if treated and an unfavorable outcome if untreated; it cannot be estimated from experimental data, even when conditioned on fine-grained features, because we cannot test both possibilities for an individual. Tian and Pearl provided bounds on this and other probabilities of causation using a combination of experimental and observational data. Those bounds, though tight, can be narrowed significantly when structural information is available in the form of a causal model. This added information may provide the power to solve central problems, such as explainable AI, legal responsibility, and personalized medicine, all of which demand counterfactual logic. This paper derives, analyzes, and characterizes these new bounds, and illustrates some of their practical applications. ## Full text ## 1 Introduction Machine learning advances have enabled tremendous capabilities of learning functions accurately and efficiently from enormous quantities of data. These functions allow for better policies, like whether a surgery, chemotherapy, or radiation therapy is most effective for a population of given characteristics such as age, sex, and type of symptoms. However, this mapping from characteristics to efficacy can be quite misleading when applied to individual decision making, even when the data originate from a randomized controlled trial (RCT). To see why let's follow the example treated in [Mueller and Pearl, 2020]. Imagine a novel vaccine for a deadly virus in the midst of a pandemic is in short supply. We want to administer the vaccine to people most likely to benefit from it. In other words, we need to identify the group most likely to both survive if vaccinated and succumb if unvaccinated. A clinical study is conducted to test the effectiveness of the vaccine. For simplicity, let's assume a binary age classification: sixty years old and under and over sixty years old. Older people survive 57% of the time when vaccinated and 37% of the time when unvaccinated, while younger people survive 55% of the time when vaccinated and 45% of the time when unvaccinated. A na¨ıve interpretation is that the vaccine is 10 percentage points more effective for older people and, therefore, they should be vaccinated first. However, a different picture emerges if we assess the percentage of beneficiaries in the two groups. These percentages, known as Probability of Necessity and Sufficiency (PNS) [Pearl, 1999], can be tightly bounded [Tian and Pearl, 2000] and falls, given the data above, between 20% and 57% for the older patients and between 10% and 55% for the younger patients. We see that it's anything but clear which group should be vaccinated first. What is more remarkable is these bounds can be narrowed significantly if data from observational studies is also available, and may even flip priority from the elderly to the young. Observational studies reflect outcomes for individuals who decide on their own whether to get vaccinated or not. In our example, one can show that the bounds for over-sixties and under-sixties may become [20%, 40%] and [40%, 55%], respectively, thus reversing the na¨ıve priorities above, and clearly show priority to vaccinate the young, not the elderly. Since Tian and Pearl [Tian and Pearl, 2000], the problem of bounding probabilities of causation was analyzed by combining only two sources of information: experimental data and observational studies, making no assumptions whatsoever about the model generating the data. This paper shows 1 that, surprisingly, knowing the structure of the causal graph allows us to narrow these bounds, despite the fact that the graph may seem redundant; i.e., we already know the causal effects. Moreover, the graph adds information about an individual, although it describes properties of the population. Knowledge of the causal structure and data allows us to narrow bounds because we can then partition bounds on subsets of covariates and mediators, obtain local bounds on the partitions, and combine the bounds. This partitioning gives us a finer-grained perspective on possible values for probabilities of causation. 1 Supplementary material is available at https://ftp.cs.ucla.edu/ pub/stat ser/r505-sup.pdf The analysis of causes of effects can now take advantage of the causal diagram. ## 2 Preliminaries and RelatedWork In this section, we review the definitions for the three aspects of causation as defined in [Pearl, 1999]. We use the causal diagrams [Pearl, 1995; Spirtes et al., 2000; Pearl, 2009; Koller and Friedman, 2009] and the language of counterfactuals in its structural model semantics, as given in [Balke and Pearl, 2013; Galles and Pearl, 1998; Halpern, 2000]. We use Yx = y to denote the counterfactual sentence "Variable Y would have the value y, had X been x." For simplicity purposes, in the rest of the paper, we use yx to denote the event Yx = y, yx′ to denote the event Yx′ = y, y′x to denote the event Yx = y′ , and y′x′ to denote the event Yx′ = y′ . For notational simplicity, we limit the discussion to binary X and Y . Extension to multi-valued variables are straightforward [Pearl, 2009]. Three prominent probabilities of causation are the following: Definition 1 (Probability of necessity (PN)). Let X and Y be two binary variables in a causal model M, let x and y stand for the propositions X = true and Y = true, respectively, and x′ and y′ for their complements. The probability of necessity is defined as the expression [Pearl, 1999] Definition 2 (Probability of sufficiency (PS)). [Pearl, 1999] Definition 3 (Probability of necessity and sufficiency (PNS)). [Pearl, 1999] PNS stands for the probability that y would respond to x both ways, and therefore measures both the sufficiency and necessity of x to produce y. Tian and Pearl [Tian and Pearl, 2000] provide tight bounds for PNS, PN, and PS without a causal diagram using Balke's program [Balke and Pearl, 1997] (we will call them TianPearl bounds). Li and Pearl [Li and Pearl, 2019] provide a theoretical proof of the tight bounds for PNS, PS, PN, and other probabilities of causation without a causal diagram. PNS, PN, and PS have the following tight bounds: Note that we only consider PNS and PN here because the bounds of PS can be easily obtained by exchanging x with x′ and y with y′ in the bounds of PN. To obtain bounds for a specific population, defined by a set C of characteristics, the expressions above should be modified by conditioning each term on C = c. In this paper, however, we obtain narrower bounds of PNS by leveraging another source of knowledge - the causal diagram behind the data, together with measurements of a set Z of covariates in that diagram. We provide graphical conditions under which the availability of such measurements would improve the bounds and demonstrate, both analytically and by simulation, the degree of improvement achieved. Narrower bounds and graphical criteria can be obtained for PN and PS through the same mechanism detailed in the proofs in the appendix. ## 3 Bounds with Causal Diagram ## 3.1 Non-descendant Covariates Theorems 4 and 5 below provide bounds for PNS when a set Z of variables can be measured which satisfy only one simple condition: Z contains no descendants of X. This condition is important because if X was set to x and Z contains a descendant of X, then Z could be altered as well and P(yx|z) would be unmeasurable. If the descendant is independent of Yx, then P(yx|z) would be measurable, but that descendant wouldn't contribute to any narrowing of bounds. These bounds are always contained within the Tian-Pearl bounds of equations 4, 5, 6, and 7. Theorem 4. Given a causal diagram G and distribution compatible with G, let Z be a set of variables that does not contain any descendant of X in G, then PNS is bounded as follows: Figure 1: Z is not a descendant of X ![Image](https://ijcai.org/proceedings/2022/fulltext/376/image_000000_5b04599c3036b61537425713f1a318a2021d1b884f8785de1c05e05496529d02.png) Figure 2: Mediator Z with direct effect ![Image](https://ijcai.org/proceedings/2022/fulltext/376/image_000001_362b2b951bfda145b31e1c03afb3f51333bf5b3056d53237f0439849ae3663c6.png) Note that, unlike the subpopulation bounds, where each term is conditioned on C = c, here PNS is not conditioned on Z = z. This is because the measurement of Z is conducted in the study, but may not be available for the individual seeking advice. Examples are illustrated in Section 4. Note also that if only experimental data are available (i.e., P (Y ), P (Y, X), P (Y |Z), P (Y, X|Z) are not measured), arguments to the max or min functions involving observational data can be disregarded. For example, the lower bounds of Theorem 4 would become max{P(yx) - P (yx′ ), ∑z max{0, P (yx|z) - P (yx′ |z)} × P (z)}. ## Sufficient Covariates In Figures 1a and 1b, Z is not a descendant of X and additionally satisfies the back-door criterion. For such cases the PNS bounds can be simplified: Theorem 5. Given a causal diagram G and distribution compatible with G, let Z be a set of variables satisfying the backdoor criterion [Pearl, 1993] in G, then the PNS is bounded as follows: The significance of Theorem 5 is due to the ability to compute bounds using purely observational data. ## 3.2 Mediation ## Partial Mediator In Figure 2, Z is a descendant of X, so we cannot use Theorems 4 and 5. However, the absence of confounders between Z and Y and between X and Y permits us to bound PNS as follows: Figure 3: Mediator Z with no direct effect ![Image](https://ijcai.org/proceedings/2022/fulltext/376/image_000002_99649aaff699b5482155bc73f4afce02efef59bd5da93ee9b7ff3a4fdb789bf9.png) Theorem 6. Given a causal diagram G and distribution compatible with G, let Z be a set of variables such that ∀x, x′ ∈ X : x ̸= x′ , (Yx ⊥⊥ X ∪ Zx′ | Zx) in G, then the PNS is bounded as follows: Note that although this lower bound is unchanged from Tian and Pearl, the upper bound contains a vital additional argument to the min function. This new term can significantly reduce the upper bound. The rest of the terms are included because sometimes Tian and Pearl's bounds are superior. The following Theorem has the same quality. ## Pure Mediator Figure 3 is a special case of Figure 2, in which X has no direct effect on Y . The resulting bounds for PNS read: Theorem 7. Given a causal diagram G in Figure 3 and distribution that compatible with G, then PNS are bounded as follow: Table 1: Results of a drug study with gender taken into account | | Drug | No Drug | |---------|--------------------------------|--------------------------------| | Women | 1 out of 110 recovered (1%) | 13 out of 120 recovered (11%) | | Men | 313 out of 354 recovered (88%) | 114 out of 116 recovered (98%) | | Overall | 314 out of 464 recovered (68%) | 127 out of 236 recovered (54%) | The core terms for Theorems 6 and 7 added to the upper bounds notably only require observational data. ## 4 Examples ## 4.1 Credit to the Treatment The manufacturer of a drug wants to claim that a non-trivial number of recovered patients who were given access to the drug owe their recovery to the drug. So they conduct an observational study; they record the recovery rates of 700 patients. 464 patients chose to take the drug and 236 patients did not. The results of the study are in table 1. The manufacturer claims success for their drug because the overall recovery rate from the observational study has increased from 54% to 68% for non-drug-takers to drug-takers. The number of recovered patients that should credit the drug for their recovery are those who would recover if they had taken the drug and would not recover if they had not taken the drug. This is the PNS. Let X = x denote the event that the patient took the drug and X = x′ denote the event that the patient did not take the drug. Let Y = y denote the event that the patient has recovered and Y = y′ denote the event that the patient has not recovered. Let Z = z represent female patients and Z = z′ represent male patients. Suppose we know an additional fact, estrogen has a negative effect on recovery, so women are less likely to recover than men, regardless of the drug. Additionally, as we can see from the data, men are significantly more likely to take the drug than women are. The causal diagram is shown in Figure 1a. Node Z on the graph satisfies the back-door criterion, therefore we can compute the causal effect P(yx) and P(y′x ) via the adjustment formula [Pearl, 1993] and observational data from table 1, where, Therefore, the bounds of PNS computed using equations 4 and 5 are 0 ≤ PNS ≤ 0.297, where the diagram was used only to identify the causal effects yx and yx′ . These bounds aren't informative enough to conclude whether or not the drug was the cause of recovery for a meaningful number of patients. They suggest that the fraction of beneficiaries can be as low as 0% or as high as 29.7%. Now, consider the bounds in Theorem 5 which takes into account the position of Z in the diagram. Since Z satisfies the back-door criterion, we can use equations 10 and 11 to compute 0 ⩽ PNS ⩽ 0.01. The conclusion now is obvious. At most 7 out of 314 patients' recoveries can be credited to the drug. This is strong evidence that counters the manufacturer's claim. ## 4.2 InflammationMediator As before, let X and Y represent drug consumption and recovery. Let Z represent acute inflammation with z being present and z′ being absent. The drug reduces inflammation. However, in some people the drug causes acute inflammation, which has adverse effects on recovery. The causal structure is depicted in Figure 3. We observe the following proportions among drug takers, non-takers, with inflammation, and without inflammation: The Tian-Pearl PNS upper bound is: Given that the lower bound is 0, these bounds are not very informative. If we knew that an individual would react to the drug with acute inflammation, we would only look at the data comprising of people reacting to the drug with acute inflammation. Since we are conditioning on z, PNS = 0 because the outcome, Y , will have the same result regardless of whether the person consumed the drug. So knowing a person's inflammation response to the drug narrows PNS from a wide [0, 0.5] to a point estimate of 0. Imagine, for this drug, that we can't know ahead of time how a person will react inflammation-wise. We can only observe acute inflammation after the drug is administered. Since we have population data from patients who have already taken the drug, we can utilize this mediator to bound the PNS for new patients who haven't yet taken the drug: The mediator-improved PNS upper bound is significantly smaller than what the Tian-Pearl upper bound provides, 0.1 vs 0.5. The new upper bound can now be effectively weighed against other factors like cost and side-effects. ## 4.3 Ancestral Covariate Let's continue from the introduction, where X represents vaccination with x being vaccinated and x′ being unvaccinated and Y represents survival with y is surviving and y′ is succumbing to the pandemic. Instead of classifying by age, let's assume our machine learning algorithm uncovers a correlation between survival and ancestry. Let Z represent ancestry and, for simplicity, there are only two ancestries, z and z′ . Either graph of Figure 1 is representative of this. Our RCT data reveals: (a) Z is a non-descendant of X ![Image](https://ijcai.org/proceedings/2022/fulltext/376/image_000003_dd848b2a5fccff92ec46197bf65f7357af7f3f4a357bb1f143b9eb0c7a7fe842.png) ![Image](https://ijcai.org/proceedings/2022/fulltext/376/image_000004_36935cd807a2053fa32f4d6f4cd97ccac6e1ce2369613fabb150e4c61cd6dfc7.png) Figure 4: Causal diagrams for simulation We now have four different bounds on PNS: Tian-Pearl =⇒ 0.1 ⩽ PNS ⩽ 0.5 Covariate-improved =⇒ 0.275 ⩽ PNS ⩽ 0.5 Person has ancestry z =⇒ 0.55 ⩽ PNS ⩽ 0.75 Person has ancestry z′ =⇒ 0 ⩽ PNS ⩽ 0.25 As expected, using the causal diagram and ancestral Z yields narrower bounds than the Tian-Pearl bounds. However, it's surprising that knowing a person has either ancestry z or z′ gives us bounds outside of our new bounds. In fact, they are completely outside the wider Tian-Pearl bounds. This is discussed in section 6. In the meantime, it's important to recognize that the last two ancestry-specific PNS bounds are what should be referred to if an individual knows their ancestry. The covariate-improved PNS bounds should only be referred to if a person's ancestry is unknown. This might be because the person was adopted with no hint as to whether they're from ancestry z or z′ (physical features are right in between or indistinguishable). ## 5 Simulation Results We randomly generated 100000 sample distributions compatible with each the causal diagrams depicted in Figures 4a, 1a, 4b, and 3 for Theorems 4, 5, 6, and 7, respectively. Given sample distribution i, let [ai, bi] be the bounds utilizing the proposed Theorems and [ci, di] be the traditional Tian-Pearl bounds [Li and Pearl, 2021]. The following is computed for each causal diagram as summarized in Table 2: - Average increased PNS lower bound: ∑(ai-ci) 100000 - Average decreased PNS upper bound: ∑(di-bi) 100000 - Average gap in Tian-Pearl PNS bounds: ∑(di-ci) 100000 - Average gap utilizing Theorems 4, 5, 6, and 7: ∑(bi-ai) 100000 - Number of sample distributions benefiting from Theorems 4, 5, 6, and 7: ∑ ei, where ei = 1 if (ai > ci) or (bi < di), ei = 0 otherwise. Table 2: Performance metrics for Theorems 4 (Non-desc), 5 (Suff covar), 6 (Part med & Part med 2), and 7 (Pure med) | | Incr'd lower bound | Decr'd upper bound | Tian-Pearl PNS gap | Theorems PNS gap | Samples benefiting | |------------|----------------------|----------------------|----------------------|--------------------|----------------------| | Non-desc | 0.0238 | 0.0237 | 0.2673 | 0.2197 | 85.622% | | Suff covar | 0.0266 | 0.0264 | 0.2197 | 0.1668 | 75.025% | | Part med | 0.0000 | 0.0047 | 0.2289 | 0.2242 | 12.532% | | Part med 2 | 0.0000 | 0.0382 | 0.2768 | 0.2386 | 100.00% | | Pure med | 0.0000 | 0.0935 | 0.2605 | 0.1670 | 100.00% | Figure 5: PNS bounds for causal diagram of Figure 4a ![Image](https://ijcai.org/proceedings/2022/fulltext/376/image_000005_0497e3b0d600a3b2c39d4448c36bd4184c2362d7e463816ca90727388cf3e490.png) For each causal diagram, 100 out of 100000 sample distributions are randomly selected, sorted by lower and upper PNS bound, and then drawn with and without considering the causal diagram (Figures 5 to 8). Table 2 shows the average gaps between Tian-Pearl PNS bounds and Theorem 6's bounds are similar for the partial mediator of Figure 4b (Part med in Table 2). This is because only 12.532% of samples are narrowed by the proposed Theorem 6. A second set of sample distributions were generated repeatedly until 100000 narrowed samples were available (Part med 2 in Table 2). This time the difference in gaps were significant, which is important if the costs of including partial mediator data are low. ## 6 Discussion We have shown that knowledge of a causal structure enables narrower PNS bounds to be estimated, compared with the tight bounds of Tian and Pearl which were derived without such knowledge. However, it must be emphasized that this narrowing is only applicable to individuals whose Z characteristics are not known at decision time. If their Z values are known, the bounds of equations 4 and 5, conditioned on those values, should be consulted. Example 4.3 provides a scenario where people who know their ancestry have very different PNS bounds than people who don't know their ancestry. You would expect the additional information of ancestral knowledge would further narrow the bounds, but they change the bounds to a different non-overlapping range. This violates the heuristic that additional information should narrow the bounds or, at worst, not widen them. To rephrase, if you don't know someone's ancestry, the probability they benefit from this drug is between 0.275 and 0.5. Once you acquire the additional information that the person is of ancestry z, the probability they benefit from this treatment becomes between 0.55 and 0.75. How is this possible? Was the person's probability of benefiting never really between 0.275 and 0.5 that we calculated before knowing their ancestry? Figure 6: PNS bounds for causal diagram of Figure 1a ![Image](https://ijcai.org/proceedings/2022/fulltext/376/image_000006_d7d03c7738b6cab42163a67a83bf0a55422ff58c287cfec386c951ae7e4bc4e4.png) Figure 7: PNS bounds for causal diagram of Figure 4b among narrowed samples referenced by part med 2 in Table 2 ![Image](https://ijcai.org/proceedings/2022/fulltext/376/image_000007_d98e797dc5e8641afe3965e2ef1de5a6204865b22f687ec6d10285935fb66b18.png) The reason for this seeming inconsistency is that we're asking different questions. When we didn't know the ancestry, we were asking, "what is the probability of benefiting for a person regardless of ancestry?" When we found out the person is of ancestry z, we then asked a different question, "what is the probability of benefiting for a person of ancestry z?" The additional information of the person's ancestry didn't help the first question and the second question isn't answerable without the additional information. The following example will illuminate the reasons for this phenomenon [Pearl, 2009, p. 296]. Let the covariate Z stand for the outcome of a fair coin toss, so P(Z = heads) = 0.5. Without knowing what treatment X and success Y represent, let's assume the following measurements are taken: Figure 8: PNS bounds for causal diagram of Figure 3 ![Image](https://ijcai.org/proceedings/2022/fulltext/376/image_000008_59eeedc81eadb4ccf15043de1e7d389f3485c1ae86ed888488792f7980f8ae9e.png) Tian-Pearl bounds gives us 0 ⩽ PNS ⩽ 0.5 and the bounds utilizing Z are 0.5 ⩽ PNS ⩽ 0.5 or PNS = 0.5. Now, let's uncover the functional mechanism, x represents betting $1 on heads, x′ represents betting $1 on tails, y represents winning $1, and y′ represents losing $1. It should now be clear why P(yx) = P(yx′ ) = 0.5. Without knowing the coin toss result, Z, the odds of winning $1 are 50/50 whether you bet on heads or tails. PNS is also 0.5 because benefiting from betting on heads is true only when the coin toss was heads. The coin toss is heads 50% of the time. This brings us back to the PNS bounds when we have the additional information of what the coin toss result was. If we know the coin toss resulted in heads, then the probability of benefiting from betting on heads is 100%. Similarly, if we know the coin toss resulted in tails, then the probability of benefiting from betting on heads is 0%. In other words PNS(heads) = 1 and PNS(tails) = 0. If the coin toss is heads, winning only happens when betting on heads. Even though the bounds are completely different when we provided with the very useful additional information of the coin toss, there is clearly no contradiction here. There was a 50% probability of benefiting from betting on heads when we didn't know the coin toss result and a 100% probability of benefiting from betting on heads when we knew the coin toss resulted in heads. We were asking two separate questions. The first question was, "what is the probability of benefiting regardless of coin toss result?" The second question was, "what is the probability of benefiting for a coin toss of heads?" ## 7 Conclusion In this work, we have developed a graphical method of learning individualized functions (representing PNS, PN, and PS) from population data, based on the structure of the causal graph. These methods generalize the PN, PS, and PNS bounds derived in [Tian and Pearl, 2000], the bounds derived in [Kuroki and Cai, 2011], and the PN bounds derived in [Dawid et al., 2017]. Often these functions return bounds rather than point estimates. This paper shows nevertheless that the bounds obtained can be quite informative. Machine learning algorithms can easily incorporate these techniques to achieve both data interpretability and decision making accuracy for situationspecific cases. ## Acknowledgments This research was supported in parts by grants from the National Science Foundation [#IIS-2106908], Office of Naval Research [#N00014-17-S-12091 and #N00014-21-1-2351], and Toyota Research Institute of North America [#PO-000897]. ## References 1. [Balke and Pearl, 1997] Alexander Balke and Judea Pearl. Probabilistic Counterfactuals: Semantics, Computation, and Applications. PhD thesis, University of California, Los Angeles, 1997. 2. [Balke and Pearl, 2013] Alexander Balke and Judea Pearl. Counterfactuals and policy analysis in structural models. arXiv preprint arXiv:1302.4929, 2013. 3. [Dawid et al., 2017] Philip Dawid, Monica Musio, and Rossella Murtas. The probability of causation. Law, Prob- ability and Risk, 16(4):163-179, 2017. 4. [Galles and Pearl, 1998] David Galles and Judea Pearl. An axiomatic characterization of causal counterfactuals. Foun- dations of Science, 3(1):151-182, 1998. 5. [Halpern, 2000] Joseph Y Halpern. Axiomatizing causal rea- soning. Journal of Artificial Intelligence Research, 12:317- 337, 2000. 6. [Koller and Friedman, 2009] Daphne Koller and Nir Fried- man. Probabilistic Graphical Models: Principles and Tech- niques. MIT press, 2009. 7. [Kuroki and Cai, 2011] Manabu Kuroki and Zhihong Cai. Statistical analysis of 'probabilities of causation' using co-variate information. Scandinavian Journal of Statistics, 38(3):564-577, 2011. 8. [Li and Pearl, 2019] Ang Li and Judea Pearl. Unit selection based on counterfactual logic. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 1793-1799. AAAI Press, 2019. 9. [Li and Pearl, 2021] Ang Li and Judea Pearl. Unit selection with causal diagram. arXiv preprint arXiv:2109.07556, 2021. 10. [Mueller and Pearl, 2020] Scott Mueller and Judea Pearl. Which Patients are in Greater Need: A counterfactual analysis with reflections on COVID-19. https://ucla.in/ 39Ey8sU+, 2020. Accessed: 2022-06-05. 11. [Pearl, 1993] Judea Pearl. Aspects of graphical models con- nected with causality. Proceedings of the 49th Session of the International Statistical Institute, Italy, pages 399-401, 1993. 12. [Pearl, 1995] Judea Pearl. Causal diagrams for empirical research. Biometrika, 82(4):669-688, 1995. 13. [Pearl, 1999] Judea Pearl. Probabilities of Causation: Three counterfactual interpretations and their identification. Syn- these, 121(1-2):93-149, 1999. 14. [Pearl, 2009] Judea Pearl. Causality. Cambridge University Press, Second edition, 2009. 15. [Spirtes et al., 2000] Peter Spirtes, Clark N Glymour, Richard Scheines, and David Heckerman. Causation, Prediction, and Search. MIT press, 2000. 16. [Tian and Pearl, 2000] Jin Tian and Judea Pearl. Probabili- ties of causation: Bounds and identification. Annals of Mathematics and Artificial Intelligence, 28(1-4):287-313, 2000.