{"type":"ScholarlyArticle","title":"Statistical Learning with a Nuisance Component (Extended Abstract)","authors":[{"name":"Dylan J. Foster","slug":"dylan-j-foster","given":"Dylan J.","family":"Foster","affiliation":"Massachusetts Institute of Technology"},{"name":"Vasilis Syrgkanis","slug":"vasilis-syrgkanis","given":"Vasilis","family":"Syrgkanis","affiliation":"Microsoft Research , New England"}],"abstract":"We provide excess risk guarantees for statistical learning in a setting where the population risk with respect to which we evaluate a target parameter depends on an unknown parameter that must be estimated from data (a \"nuisance parameter\"). We analyze a two-stage sample splitting meta-algorithm that takes as input two arbitrary estimation algorithms: one for the target parameter and one for the nuisance parameter. We show that if the population risk satisfies a condition called Neyman orthogonality, the impact of the nuisance estimation error on the excess risk bound achieved by the meta-algorithm is of second order. Our theorem is agnostic to the particular algorithms used for the target and nuisance and only makes an assumption on their individual performance. This enables the use of a plethora of existing results from statistical learning and machine learning literature to give new guarantees for learning with a nuisance component. Moreover, by focusing on excess risk rather than parameter estimation, we can give guarantees under weaker assumptions than in previous works and accommodate the case where the target parameter belongs to a complex nonparametric class. We characterize conditions on the metric entropy such that oracle rates---rates of the same order as if we knew the nuisance parameter---are achieved. We also analyze the rates achieved by specific estimation algorithms such as variance-penalized empirical risk minimization, neural network estimation and sparse high-dimensional linear model estimation. We highlight the applicability of our results in four settings of central importance in the literature: 1) heterogeneous treatment effect estimation, 2) offline policy optimization, 3) domain adaptation, and 4) learning with missing data.","doi":"10.24963/ijcai.2020/654","url":"https://ijcai.org/proceedings/2020/654","pdf":"https://ijcai.org/proceedings/2020/0654.pdf","markdown":"https://ijcai.org/proceedings/2020/654.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/654","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Sister Conferences Best Papers","track":"Sister Conferences Best Papers","number":654,"pages":{"first":4726,"last":4729},"keywords":["Machine Learning: Learning Theory"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25680/file_25680.m3u8","length_sec":297,"provider":"hls"}],"references":[{"position":1,"raw":"[Athey and Wager, 2017] Susan Athey and Stefan Wager. Ef- ficient policy learning. arXiv preprint arXiv:1702.02896, 2017.","title":"Efficient policy learning","authors":"Susan Athey, Stefan Wager","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Athey et al., 2019] Susan Athey, Julie Tibshirani, and Ste- fan Wager. Generalized random forests. The Annals of Statistics, 47(2):1148-1178, 2019.","title":"Generalized random forests","authors":"Susan Athey, Julie Tibshirani, Stefan Wager","year":"2019","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bartlett et al., 2005] Peter L Bartlett, Olivier Bousquet, and Shahar Mendelson. Local rademacher complexities. The Annals of Statistics, 33(4):1497-1537, 2005.","title":"Local rademacher complexities","authors":"Olivier Peter L Bartlett, Shahar Bousquet, Mendelson","year":"2005","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bousquet et al., 2004] Olivier Bousquet, Stéphane Boucheron, and Gábor Lugosi. Introduction to sta- tistical learning theory. In Advanced lectures on machine learning, pages 169-207. Springer, 2004.","title":"Introduction to sta-tistical learning theory","authors":"Olivier Bousquet, Stéphane Boucheron, Gábor Lugosi","year":"2004","venue":"Advanced lectures on machine learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chernozhukov et al., 2016] Victor Chernozhukov, Juan Car- los Escanciano, Hidehiko Ichimura, and Whitney K Newey. Locally robust semiparametric estimation. arXiv preprint arXiv:1608.00033, 2016.","title":"Locally robust semiparametric estimation","authors":"Victor Chernozhukov, Juan Carlos Escanciano, Hidehiko Ichimura, Whitney K Newey","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chernozhukov et al., 2017] Victor Chernozhukov, Matt Goldman, Vira Semenova, and Matt Taddy. Orthogonal machine learning for demand estimation: High dimen- sional causal inference in dynamic panels. arXiv preprint arXiv:1712.09988, 2017.","title":"Orthogonal machine learning for demand estimation: High dimensional causal inference in dynamic panels","authors":"Victor Chernozhukov, Matt Goldman, Vira Semenova, Matt Taddy","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chernozhukov et al., 2018a] Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased ma- chine learning for treatment and structural parameters. The Econometrics Journal, 21(1):C1-C68, 2018.","title":"Double/debiased machine learning for treatment and structural parameters","authors":"Chernozhukov","year":"2018","venue":"The Econometrics Journal","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chernozhukov et al., 2018b] Victor Chernozhukov, Denis Nekipelov, Vira Semenova, and Vasilis Syrgkanis. Plug- in regularized estimation of high-dimensional parame- ters in nonlinear semiparametric models. arXiv preprint arXiv:1806.04823, abs/1806.04823, 2018.","title":"Plugin regularized estimation of high-dimensional parameters in nonlinear semiparametric models","authors":"Chernozhukov","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chernozhukov et al., 2018c] Victor Chernozhukov, Whitney Newey, and James Robins. Double/de-biased machine learning using regularized riesz representers. arXiv preprint arXiv:1802.08667, 2018.","title":"Double/de-biased machine learning using regularized riesz representers","authors":"Chernozhukov","year":"2018","venue":"Victor Chernozhukov","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dudík et al., 2011] Miroslav Dudík, John Langford, and Li- hong Li. Doubly robust policy evaluation and learning. In Proceedings of the 28th International Conference on In- ternational Conference on Machine Learning, pages 1097- 1104. Omnipress, 2011.","title":"Doubly robust policy evaluation and learning","authors":"Dudík","year":"2011","venue":"Proceedings of the 28th International Conference on International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Foster and Syrgkanis, 2019] Dylan J. Foster and Vasilis Syrgkanis. Statistical learning with a nuisance component. In Alina Beygelzimer and Daniel Hsu, editors, Proceed- ings of the Thirty-Second Conference on Learning Theory, volume 99 of Proceedings of Machine Learning Research, pages 1346-1348, Phoenix, USA, 25-28 Jun 2019. PMLR.","title":"Statistical learning with a nuisance component","authors":"Dylan J Foster, Vasilis Syrgkanis","year":"2019","venue":"Proceedings of the Thirty-Second Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Friedberg et al., 2018] Rina Friedberg, Julie Tibshirani, Su- san Athey, and Stefan Wager. Local linear forests. arXiv preprint arXiv:1807.11408, 2018.","title":"Local linear forests","authors":"Rina Friedberg, Julie Tibshirani, Susan Athey, Stefan Wager","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kallus and Zhou, 2018] Nathan Kallus and Angela Zhou. Policy evaluation and optimization with continuous treat- ments. In International Conference on Artificial Intelli- gence and Statistics, pages 1243-1251, 2018.","title":"Policy evaluation and optimization with continuous treatments","authors":"Zhou Kallus, Nathan Kallus, Angela Zhou","year":"2018","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Koltchinskii and Panchenko, 2000] V. Koltchinskii and D. Panchenko. Rademacher processes and bounding the risk of function learning. High Dimensional Probability II, 47:443-459, 2000.","title":"Rademacher processes and bounding the risk of function learning","authors":"V Koltchinskii, D Panchenko","year":"2000","venue":"High Dimensional Probability II","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kosorok, 2008] Michael R Kosorok. Introduction to empir- ical processes and semiparametric inference. Springer, 2008.","title":"Introduction to empirical processes and semiparametric inference","authors":"Michael R Kosorok","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Maurer and Pontil, 2009] Andreas Maurer and Massimil- iano Pontil. Empirical bernstein bounds and sample vari- ance penalization. In The 22nd Conference on Learning Theory (COLT, 2009.","title":"Andreas Maurer and Massimiliano Pontil. Empirical bernstein bounds and sample variance penalization","authors":"Pontil Maurer","year":"2009","venue":"The 22nd Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Neyman, 1959] Jerzy Neyman. Optimal asymptotic tests of composite hypotheses. Probability and statsitics, pages 213-234, 1959.","title":"Optimal asymptotic tests of composite hypotheses","authors":"Jerzy Neyman, Neyman","year":"1959","venue":"Probability and statsitics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Neyman, 1979] Jerzy Neyman. C (α) tests and their use. Sankhyā: The Indian Journal of Statistics, Series A, pages 1-21, 1979.","title":"C (α) tests and their use","authors":"Jerzy Neyman","year":"1979","venue":"Sankhyā: The Indian Journal of Statistics, Series A","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Nie and Wager, 2017] Xinkun Nie and Stefan Wager. Quasi- oracle estimation of heterogeneous treatment effects. arXiv preprint arXiv:1712.04912, 2017.","title":"Xinkun Nie and Stefan Wager. Quasioracle estimation of heterogeneous treatment effects","authors":"Wager Nie","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Oprescu et al., 2019] Miruna Oprescu, Vasilis Syrgkanis, and Zhiwei Steven Wu. Orthogonal random forest for causal inference. In International Conference on Machine Learning, pages 4932-4941, 2019.","title":"Orthogonal random forest for causal inference","authors":"Miruna Oprescu, Vasilis Syrgkanis, Zhiwei Steven Wu","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rakhlin et al., 2017] Alexander Rakhlin, Karthik Sridharan, and Alexandre B Tsybakov. Empirical entropy, minimax regret and minimax risk. Bernoulli, 23(2):789-824, 2017.","title":"Empirical entropy, minimax regret and minimax risk","authors":"Alexander Rakhlin, Karthik Sridharan, Alexandre B Tsybakov","year":"2017","venue":"Bernoulli","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Robinson, 1988] Peter M Robinson. Root-n-consistent semi- parametric regression. Econometrica: Journal of the Econo- metric Society, pages 931-954, 1988.","title":"Root-n-consistent semiparametric regression","authors":"Peter M Robinson","year":"1988","venue":"Econometrica: Journal of the Econometric Society","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Swaminathan and Joachims, 2015] Adith Swaminathan and Thorsten Joachims. Counterfactual risk minimization: Learning from logged bandit feedback. In International Conference on Machine Learning, pages 814-823, 2015.","title":"Counterfactual risk minimization: Learning from logged bandit feedback","authors":"Joachims Swaminathan, Adith Swaminathan, Thorsten Joachims","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vapnik, 1995] Vladimir N Vapnik. The nature of statistical learning theory. Springer, 1995.","title":"The nature of statistical learning theory","authors":"N Vladimir, Vapnik","year":"1995","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yang and Barron, 1999] Yuhong Yang and Andrew Barron. Information-theoretic determination of minimax rates of convergence. Annals of Statistics, pages 1564-1599, 1999.","title":"Information-theoretic determination of minimax rates of convergence","authors":"Yang, Barron ;, Yuhong Yang, Andrew Barron","year":"1999","venue":"Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhou et al., 2018] Zhengyuan Zhou, Susan Athey, and Stefan Wager. Offline multi-action policy learn- ing: Generalization and optimization. arXiv preprint arXiv:arXiv:1810.04778, 2018.","title":"Offline multi-action policy learning: Generalization and optimization","authors":"Zhengyuan Zhou, Susan Athey, Stefan Wager","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}