{"type":"ScholarlyArticle","title":"What to Expect of Classifiers? Reasoning about Logistic Regression with Missing Features","authors":[{"name":"Pasha Khosravi","slug":"pasha-khosravi","given":"Pasha","family":"Khosravi","affiliation":"University of California , Los Angeles"},{"name":"Yitao Liang","slug":"yitao-liang","given":"Yitao","family":"Liang","affiliation":"University of California , Los Angeles"},{"name":"YooJung Choi","slug":"yoojung-choi","given":"YooJung","family":"Choi","affiliation":"University of California , Los Angeles"},{"name":"Guy Van den Broeck","slug":"guy-van-den-broeck","given":"Guy","family":"Van den Broeck","affiliation":"University of California , Los Angeles"}],"abstract":"While discriminative classifiers often yield strong predictive performance, missing feature values at prediction time can still be a challenge. Classifiers may not behave as expected under certain ways of substituting the missing values, since they inherently make assumptions about the data distribution they were trained on. In this paper, we propose a novel framework that classifies examples with missing features by computing the expected prediction with respect to a feature distribution. Moreover, we use geometric programming to learn a naive Bayes distribution that embeds a given logistic regression classifier and can efficiently take its expected predictions. Empirical evaluations show that our model achieves the same performance as the logistic regression with all features observed, and outperforms standard imputation techniques when features go missing during prediction time. Furthermore, we demonstrate that our method can be used to generate ``sufficient explanations'' of logistic regression classifications, by removing features that do not affect the classification.","doi":"10.24963/ijcai.2019/377","url":"https://ijcai.org/proceedings/2019/377","pdf":"https://ijcai.org/proceedings/2019/0377.pdf","markdown":"https://ijcai.org/proceedings/2019/377.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/377","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":377,"pages":{"first":2716,"last":2724},"keywords":["Uncertainty in AI: Graphical Models","Machine Learning: Classification","Machine Learning: Learning Generative Models","Uncertainty in AI: Bayesian Networks"],"videos":[],"references":[{"position":1,"raw":"[Azur et al., 2011] Melissa J Azur, Elizabeth A Stuart, Con- stantine Frangakis, and Philip J Leaf. Multiple imputation by chained equations: what is it and how does it work? International journal of methods in psychiatric research, 2011.","title":"Multiple imputation by chained equations: what is it and how does it work","authors":"Melissa J Azur, Elizabeth A Stuart, Constantine Frangakis, Philip J Leaf","year":"2011","venue":"International journal of methods in psychiatric research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Batista et al., 2002] Gustavo EAPA Batista, Maria Carolina Monard, et al. A study of k-nearest neighbour as an impu- tation method. HIS, 2002.","title":"A study of k-nearest neighbour as an imputation method","authors":"Gustavo Eapa Batista, Maria Carolina Monard","year":"2002","venue":"HIS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Blackard and Dean, 1999] Jock A Blackard and Denis J Dean. Comparative accuracies of artificial neural networks and discriminant analysis in predicting forest cover types from cartographic variables. Computers and electronics in agriculture, 1999.","title":"Comparative accuracies of artificial neural networks and discriminant analysis in predicting forest cover types from cartographic variables","authors":"Jock A Blackard, Denis J Dean","year":"1999","venue":"Computers and electronics in agriculture","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bouchard and Triggs, 2004] Guillaume Bouchard and Bill Triggs. The tradeoff between generative and discrimina- tive classifiers. In COMPSTAT, 2004.","title":"The tradeoff between generative and discriminative classifiers","authors":"Triggs Bouchard, Guillaume Bouchard, Bill Triggs","year":"2004","venue":"COMPSTAT","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Boyd et al., 2007] Stephen Boyd, Seung-Jean Kim, Lieven Vandenberghe, and Arash Hassibi. A tutorial on geometric programming. Optimization and engineering, 2007.","title":"A tutorial on geometric programming","authors":"Stephen Boyd, Seung-Jean Kim, Lieven Vandenberghe, Arash Hassibi","year":"2007","venue":"Optimization and engineering","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Buuren and Groothuis-Oudshoorn, 2010] Stef van Buuren and Karin Groothuis-Oudshoorn. MICE: Multivariate im- putation by chained equations in R. Journal of statistical software, 2010.","title":"MICE: Multivariate imputation by chained equations in R","authors":"Groothuis-Oudshoorn ; Stef Buuren, Karin Van Buuren, Groothuis-Oudshoorn","year":"2010","venue":"Journal of statistical software","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chang et al., 2019] Chun-Hao Chang, Elliot Creager, Anna Goldenberg, and David Duvenaud. Explaining image clas- sifiers by counterfactual generation. In ICLR, 2019.","title":"Explaining image classifiers by counterfactual generation","authors":"Chun-Hao Chang, Elliot Creager, Anna Goldenberg, David Duvenaud","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen et al., 2013] Suming Jeremiah Chen, Arthur Choi, and Adnan Darwiche. An exact algorithm for computing the same-decision probability. In IJCAI, 2013.","title":"An exact algorithm for computing the same-decision probability","authors":"Chen","year":"2013","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chen et al., 2018] Jianbo Chen, Le Song, Martin Wain- wright, and Michael Jordan. Learning to explain: An information-theoretic perspective on model interpretation. In ICML, 2018.","title":"Learning to explain: An information-theoretic perspective on model interpretation","authors":"Jianbo Chen, Le Song, Martin Wainwright, Michael Jordan","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Costa and et al., 2018] Adriana Fonseca Costa and et al. Missing data imputation via denoising autoencoders: the untold story. In Int'l Symposium on Intelligent Data Anal- ysis, 2018.","title":"Missing data imputation via denoising autoencoders: the untold story","authors":"Adriana Fonseca, Costa","year":"2018","venue":"Int'l Symposium on Intelligent Data Analysis","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Darwiche, 2009] Adnan Darwiche. Modeling and reason- ing with Bayesian networks. Cambridge University Press, 2009.","title":"Modeling and reasoning with Bayesian networks","authors":"Adnan Darwiche","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Dechter, 2013] Rina Dechter. Reasoning with probabilis- tic and deterministic graphical models: Exact algorithms. Synthesis Lectures on Artificial Intelligence and Machine Learning, 2013.","title":"Reasoning with probabilistic and deterministic graphical models: Exact algorithms","authors":"Rina Dechter","year":"2013","venue":"Synthesis Lectures on Artificial Intelligence and Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Dekel and Shamir, 2008] Ofer Dekel and Ohad Shamir. Learning to classify with missing and corrupted features. In ICML, 2008.","title":"Learning to classify with missing and corrupted features","authors":"Shamir Dekel, Ofer Dekel, Ohad Shamir","year":"2008","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Dempster et al., 1977] Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incom- plete data via the EM algorithm. Journal of the Royal Sta- tistical Society: Series B (Methodological), 1977.","title":"Maximum likelihood from incomplete data via the EM algorithm","authors":"Nan M Arthur P Dempster, Donald B Laird, Rubin","year":"1977","venue":"Journal of the Royal Statistical Society: Series B (Methodological","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Dua and Karra Taniskidou, 2017] Dheeru Dua and Efi Karra Taniskidou. UCI machine learning repository, 2017.","title":"Dheeru Dua and Efi Karra Taniskidou. UCI machine learning repository","authors":"Karra Dua, Taniskidou","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Globerson and Roweis, 2006] Amir Globerson and Sam Roweis. Nightmare at test time: Robust learning by feature deletion. In ICML, 2006.","title":"Nightmare at test time: Robust learning by feature deletion","authors":"Roweis Globerson, Amir Globerson, Sam Roweis","year":"2006","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Gondara and Wang, 2018] Lovedeep Gondara and Ke Wang. Mida: Multiple imputation using denois- ing autoencoders. In SIGKDD, 2018.","title":"Lovedeep Gondara and Ke Wang. Mida: Multiple imputation using denoising autoencoders","authors":"Wang Gondara","year":"2018","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Graham, 2012] John W Graham. Missing data: Analysis and design. Springer Science & Business Media, 2012.","title":"Missing data: Analysis and design","authors":"John W, Graham","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Jaakkola and Haussler, 1999] Tommi Jaakkola and David Haussler. Exploiting generative models in discriminative classifiers. In NeurIPS, 1999.","title":"Exploiting generative models in discriminative classifiers","authors":"Haussler ; Jaakkola, Tommi Jaakkola, David Haussler","year":"1999","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Koller and Friedman, 2009] Daphne Koller and Nir Fried- man. Probabilistic graphical models: principles and tech- niques. MIT press, 2009.","title":"Probabilistic graphical models: principles and techniques","authors":"Friedman Koller","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Liang and Van den Broeck, 2019] Yitao Liang and Guy Van den Broeck. Learning logistic circuits. In AAAI, 2019.","title":"Learning logistic circuits","authors":"Van Liang, Yitao Den Broeck, Guy Liang, Van Den Broeck","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Little and Rubin, 2014] Roderick JA Little and Donald B Rubin. Statistical analysis with missing data. 2014.","title":"Statistical analysis with missing data","authors":"J A Roderick, Donald B Little, Rubin","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Lundberg and Lee, 2017] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In NeurIPS, 2017.","title":"A unified approach to interpreting model predictions","authors":"M Scott, Su-In Lundberg, Lee","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Mattei and Frellsen, 2019] Pierre-Alexandre Mattei and Jes Frellsen. MIWAE: Deep generative modelling and impu- tation of incomplete data sets. In ICML, 2019.","title":"MIWAE: Deep generative modelling and imputation of incomplete data sets","authors":"Pierre-Alexandre Mattei, Jes Frellsen","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[McKnight et al., 2007] Patrick E McKnight, Katherine M McKnight, Souraya Sidani, and Aurelio Jose Figueredo. Missing data: A gentle introduction. Guilford Press, 2007.","title":"Missing data: A gentle introduction","authors":"Katherine M Patrick E Mcknight, Souraya Mcknight, Aurelio Jose Sidani, Figueredo","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Ng and Jordan, 2002] Andrew Y Ng and Michael I Jordan. On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes. In NeurIPS, 2002.","title":"On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes","authors":"Y Andrew, Michael I Ng, Jordan","year":"2002","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Noordewier et al., 1991] Michiel O Noordewier, Geof- frey G Towell, and Jude W Shavlik. Training knowledge- based neural networks to recognize genes in dna se- quences. In NeurIPS, 1991.","title":"Training knowledgebased neural networks to recognize genes in dna sequences","authors":"Geoffrey G Michiel O Noordewier, Jude W Towell, Shavlik","year":"1991","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Ribeiro et al., 2016] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should I trust you?: Explain- ing the predictions of any classifier. In SIGKDD, 2016.","title":"Why should I trust you?: Explaining the predictions of any classifier","authors":"Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin","year":"2016","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Roos et al., 2005] Teemu Roos, Hannes Wettig, Peter Grünwald, Petri Myllymäki, and Henry Tirri. On discrimi- native bayesian network classifiers and logistic regression. Machine Learning, 59(3), Jun 2005.","title":"On discriminative bayesian network classifiers and logistic regression","authors":"Roos","year":"2005","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Roth, 1996] Dan Roth. On the hardness of approximate rea- soning. Artificial Intelligence, 1996.","title":"On the hardness of approximate reasoning","authors":"Dan Roth","year":"1996","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Schafer, 1999] Joseph L Schafer. Multiple imputation: a primer. Statistical methods in medical research, 1999.","title":"Multiple imputation: a primer","authors":"L Joseph, Schafer","year":"1999","venue":"Statistical methods in medical research","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Sharpe and Solly, 1995] Peter K. Sharpe and RJ Solly. Dealing with missing values in neural network-based diag- nostic systems. Neural Computing & Applications, 1995.","title":"Dealing with missing values in neural network-based diagnostic systems","authors":"K Peter, Sharpe, Solly","year":"1995","venue":"Neural Computing & Applications","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Shrikumar et al., 2017] Avanti Shrikumar, Peyton Green- side, and Anshul Kundaje. Learning important features through propagating activation differences. In ICML, 2017.","title":"Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences","authors":"Shrikumar","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Sutton and McCallum, 2012] Charles Sutton and Andrew McCallum. An introduction to conditional random fields. Foundations and Trends in Machine Learning, (4), 2012.","title":"An introduction to conditional random fields","authors":"Sutton, Mccallum, Charles Sutton, Andrew Mccallum","year":"2012","venue":"Foundations and Trends in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Ulusoy and Bishop, 2005] Ilkay Ulusoy and Christopher M Bishop. Generative versus discriminative methods for ob- ject recognition. In Proceddings of CVPR, 2005.","title":"Generative versus discriminative methods for object recognition","authors":"Ilkay Ulusoy, Christopher M Bishop","year":"2005","venue":"Proceddings of CVPR","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Xia et al., 2017] Jing Xia, Shengyu Zhang, Guolong Cai, Li Li, Qing Pan, Jing Yan, and Gangmin Ning. Adjusted weight voting algorithm for random forests in handling missing values. Pattern Recognition, 2017.","title":"Adjusted weight voting algorithm for random forests in handling missing values","authors":"Jing Xia, Shengyu Zhang, Guolong Cai, Li Li, Qing Pan, Jing Yan, Gangmin Ning","year":"2017","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Xiao et al., 2017] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. 2017.","title":"Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms","authors":"Han Xiao, Kashif Rasul, Roland Vollgraf","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Yann et al., 2009] LeCun Yann, Cortes Corinna, and Christopher JC Burges. The MNIST database of hand- written digits, 2009.","title":"The MNIST database of handwritten digits","authors":"Lecun Yann, Corinna Cortes, J C Christopher, Burges","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}