{"type":"ScholarlyArticle","title":"Estimation with Incomplete Data: The Linear Case","authors":[{"name":"Karthika Mohan","slug":"karthika-mohan","given":"Karthika","family":"Mohan","affiliation":"University of California , Berkeley"},{"name":"Felix Thoemmes","slug":"felix-thoemmes","given":"Felix","family":"Thoemmes","affiliation":"Cornell University"},{"name":"Judea Pearl","slug":"judea-pearl","given":"Judea","family":"Pearl","affiliation":"University of California , Los Angeles"}],"abstract":"Traditional methods for handling incomplete data, including Multiple Imputation and Maximum Likelihood, require that the data be Missing At Random (MAR). In most cases, however, missingness in a variable depends on the underlying value of that variable. In this work, we devise model-based methods to consistently estimate mean, variance and covariance given data that are Missing Not At Random (MNAR). While previous work on MNAR data require variables to be discrete, we extend the analysis to continuous variables drawn from Gaussian distributions. We demonstrate the merits of our techniques by comparing it empirically to state of the art software packages.","doi":"10.24963/ijcai.2018/705","url":"https://ijcai.org/proceedings/2018/705","pdf":"https://ijcai.org/proceedings/2018/0705.pdf","markdown":"https://ijcai.org/proceedings/2018/705.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/705","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Uncertainty in AI","track":"Main track","number":705,"pages":{"first":5082,"last":5088},"keywords":["Uncertainty in AI: Graphical Models","Uncertainty in AI: Bayesian Networks"],"videos":[],"references":[{"position":1,"raw":"[Adams, 2007] Jon Adams. Researching Complementary and Alternative Medicine. Routledge, 2007.","title":"Researching Complementary and Alternative Medicine","authors":"Adams ;, Jon Adams","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Allison, 2003] Paul D. Allison. Missing data techniques for structural equation modeling. Journal of Abnormal Psy- chology, 112(4):545, 2003.","title":"Missing data techniques for structural equation modeling","authors":"Paul D Allison","year":"2003","venue":"Journal of Abnormal Psychology","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brito, 2004] Carlos Brito. Graphical Models for Identifica- tion in Structural Equation Models. PhD thesis, University of California Los Angeles, 2004.","title":"Graphical Models for Identification in Structural Equation Models","authors":"Carlos Brito","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Daniel et al., 2012] Rhian M. Daniel, Michael G. Kenward, Simon N. Cousens, and Bianca L. De Stavola. Using causal diagrams to guide analysis in missing data prob- lems. Statistical Methods in Medical Research, 21(3):243- 256, 2012.","title":"Using causal diagrams to guide analysis in missing data problems","authors":"M Rhian, Michael G Daniel, Simon N Kenward, Bianca L De Cousens, Stavola","year":"2012","venue":"Statistical Methods in Medical Research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Darwiche, 2009] Adnan Darwiche. Modeling and Reason- ing with Bayesian Networks. Cambridge University Press, 2009.","title":"Modeling and Reasoning with Bayesian Networks","authors":"Adnan Darwiche","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Enders, 2006] Craig K. Enders. Analyzing structural equa- tion models with missing data. In Gregory R. Hancock and Ralph O. Mueller, editors, Structural Equation Mod- eling: A Second Course, pages 315-344. Information Age Publishing, Inc., 2006.","title":"Analyzing structural equation models with missing data","authors":"K Craig, Enders","year":"2006","venue":"Structural Equation Modeling: A Second Course","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Graham, 2003] John W. Graham. Adding missing-data- relevant variables to FIML-based structural equation mod- els. Structural Equation Modeling, 10(1):80-100, 2003.","title":"Adding missing-datarelevant variables to FIML-based structural equation models","authors":"John W Graham","year":"2003","venue":"Structural Equation Modeling","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Greenland and Finkle, 1995] Sander Greenland and William D. Finkle. A critical look at methods for handling missing covariates in epidemiologic regression analyses. American Journal of Epidemiology, 142(12):1255-1264, 1995.","title":"A critical look at methods for handling missing covariates in epidemiologic regression analyses","authors":"Sander Greenland, William D Finkle","year":"1995","venue":"American Journal of Epidemiology","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Koller and Friedman, 2009] Daphne Koller and Nir Fried- man. Probabilistic Graphical Models: Principles and Techniques. The MIT Press, 2009.","title":"Probabilistic Graphical Models: Principles and Techniques","authors":"Friedman Koller","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Little, 1995] Roderick J. A. Little. Modeling the drop- out mechanism in repeated-measures studies. Journal of the American Statistical Association, 90(431):1112-1121, 1995.","title":"Modeling the dropout mechanism in repeated-measures studies","authors":"J A Roderick, Little","year":"1995","venue":"Journal of the American Statistical Association","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mohan and Pearl, 2014a] Karthika Mohan and Judea Pearl. Graphical models for recovering probabilistic and causal queries from missing data. In C. Cortes M. Welling, Z. Ghahramani and N. Lawrence, editors, Advances in Neural Information Processing Systems 27, pages 1520- 1528.","title":"Graphical models for recovering probabilistic and causal queries from missing data","authors":"Mohan, Pearl ;, Karthika Mohan, Judea Pearl","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mohan and Pearl, 2014b] Karthika Mohan and Judea Pearl. On the testability of models with missing data. In S. Kaski and J. Corander, editors, Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics (AISTATS), volume 33, pages 643-650, 2014.","title":"On the testability of models with missing data","authors":"Mohan, Pearl ;, Karthika Mohan, Judea Pearl","year":"2014","venue":"Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics (AISTATS)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mohan and Pearl, 2018] Karthika Mohan and Judea Pearl. Graphical models for processing missing data. arXiv preprint arXiv:1801.03583, 2018.","title":"Graphical models for processing missing data","authors":"Mohan, Pearl ;, Karthika Mohan, Judea Pearl","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mohan et al., 2013] Karthika Mohan, Judea Pearl, and Jin Tian. Graphical models for inference with missing data. In C.J.C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Infor- mation Processing Systems 26, pages 1277-1285.","title":"Graphical models for inference with missing data","authors":"Karthika Mohan, Judea Pearl, Jin Tian","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Osborne, 2014] Jason W. Osborne. Best Practices in Logis- tic Regression. SAGE Publications, 2014.","title":"Best Practices in Logistic Regression","authors":"Jason W Osborne","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pearl, 2009] Judea Pearl. Causality: Models, Reasoning and Inference. Cambridge Univ Press, New York, 2nd edi- tion, 2009.","title":"Judea Pearl. Causality: Models, Reasoning and Inference","authors":"Pearl","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pearl, 2013] Judea Pearl. Linear models: A useful \"micro- scope\" for causal analysis. Journal of Causal Inference, 1(1):155-170, 2013.","title":"Judea Pearl. Linear models: A useful \"microscope\" for causal analysis","authors":"Pearl","year":"2013","venue":"Journal of Causal Inference","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rubin, 1976] Donald B. Rubin. Inference and missing data. Biometrika, 63:581-592, 1976.","title":"Inference and missing data","authors":"Donald B Rubin","year":"1976","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Schlomer et al., 2010] Gabriel L. Schlomer, Sheri Bauman, and Noel A. Card. Best practices for missing data man- agement in counseling psychology. Journal of Counseling Psychology, 57(1):1, 2010.","title":"Best practices for missing data management in counseling psychology","authors":"L Gabriel, Sheri Schlomer, Noel A Bauman, Card","year":"2010","venue":"Journal of Counseling Psychology","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Schminkey et al., 2016] Donna L. Schminkey, Timo von Oertzen, and Linda Bullock. Handling missing data with multilevel structural equation modeling and full informa- tion maximum likelihood techniques. Research in Nursing & Health, 39(4):286-297, 2016.","title":"Handling missing data with multilevel structural equation modeling and full information maximum likelihood techniques","authors":"Donna L Schminkey, Linda Timo Von Oertzen, Bullock","year":"2016","venue":"Research in Nursing & Health","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shpitser et al., 2015] Iyla Shpitser, Karthika Mohan, and Judea Pearl. Missing data as a causal and probabilistic problem. In M. Meila and T. Heskes, editors, Proceedings of the Thirty-First Conference on Uncertainty in Artificial Intelligence, pages 802-811. AUAI Press, 2015.","title":"Missing data as a causal and probabilistic problem","authors":"Iyla Shpitser, Karthika Mohan, Judea Pearl","year":"2015","venue":"Proceedings of the Thirty-First Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sverdlov, 2015] Oleksandr Sverdlov. Modern Adaptive Randomized Clinical Trials: Statistical and Practical As- pects. Chapman and Hall/CRC, 2015.","title":"Modern Adaptive Randomized Clinical Trials: Statistical and Practical Aspects","authors":"Oleksandr Sverdlov, Sverdlov","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Thoemmes and Mohan, 2015] Felix Thoemmes and Karthika Mohan. Graphical representation of miss- ing data problems. Structural Equation Modeling: A Multidisciplinary Journal, 22:631-642, 2015.","title":"Felix Thoemmes and Karthika Mohan. Graphical representation of missing data problems","authors":"Mohan Thoemmes","year":"2015","venue":"Structural Equation Modeling: A Multidisciplinary Journal","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ullman and Bentler, 2003] Jodie B. Ullman and Peter M. Bentler. Structural Equation Modeling. Wiley Online Li- brary, 2003.","title":"Structural Equation Modeling","authors":"Jodie B Ullman, Peter M Bentler","year":"2003","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[van Stein and Kowalczyk, 2016] Bas van Stein and Wojtek Kowalczyk. An incremental algorithm for repairing train- ing sets with missing values. In International Confer- ence on Information Processing and Management of Un- certainty in Knowledge-Based Systems, pages 175-186. Springer, 2016.","title":"An incremental algorithm for repairing training sets with missing values","authors":"Kowalczyk ; Bas Van Stein, Wojtek Van Stein, Kowalczyk","year":"2016","venue":"International Conference on Information Processing and Management of Uncertainty in Knowledge-Based Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wright, 1921] Sewall Wright. Correlation and causation. Journal of Agricultural Research, 20(7):557-585, 1921.","title":"Correlation and causation","authors":"Wright ;, Sewall Wright","year":"1921","venue":"Journal of Agricultural Research","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}