{"type":"ScholarlyArticle","title":"Unbiased Active Semi-supervised Binary Classification Models","authors":[{"name":"JooChul Lee","slug":"joochul-lee","given":"JooChul","family":"Lee","affiliation":"Department of Mathematics and Statistics , Auburn University , USA"},{"name":"Weidong Ma","slug":"weidong-ma","given":"Weidong","family":"Ma","affiliation":"Department of Biostatistics, Epidemiology and Informatics , University of Pennsylvania Perelman School of Medicine , USA"},{"name":"Ziyang Wang","slug":"ziyang-wang","given":"Ziyang","family":"Wang","affiliation":"Department of Statistics , University of Connecticut , USA"}],"abstract":"Active learning is known to be a well-motivated algorithm that aims to maximize model performance with relatively small data, but it introduces sampling bias due to active selection. To adjust the bias, current literature utilizes corrective weights in a supervised learning approach. However, those methods consider only a small amount of actively sampled data and thus estimation efficiency can be improved using unsampled data together. In this paper, we develop an actively improved augmented estimation equation (AI-AEE) based on corrective weights as well as imputation models that allow us to leverage unlabeled data. The asymptotic distribution of the proposed estimator as the solution to the AI-AEE is derived, and an optimal sampling scheme to minimize the asymptotic mean squared error of the estimator is proposed. We then propose a general practical algorithm for training prediction models in the active and semi-supervised learning framework. The superiority of our method is demonstrated on synthetic and real data examples.","doi":"10.24963/ijcai.2024/485","url":"https://ijcai.org/proceedings/2024/485","pdf":"https://ijcai.org/proceedings/2024/0485.pdf","markdown":"https://ijcai.org/proceedings/2024/485.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/485","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":485,"pages":{"first":4389,"last":4397},"keywords":["Machine Learning: ML: Semi-supervised learning","Machine Learning: ML: Regression","Machine Learning: ML: Active learning"],"videos":[],"references":[{"position":1,"raw":"[Ai et al., 2018] Mingyao Ai, Jun Yu, Huiming Zhang, and HaiYing Wang. Optimal subsampling algorithms for big data regressions. arXiv preprint arXiv:1806.06761, 2018.","title":"Optimal subsampling algorithms for big data regressions","authors":"Ai","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cai et al., 2022] Tianxi Cai, Mengyan Li, and Molei Liu. Semi-supervised triply robust inductive transfer learning. arXiv preprint arXiv:2209.04977, 2022.","title":"Semi-supervised triply robust inductive transfer learning","authors":"Tianxi Cai, Mengyan Li, Molei Liu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cao et al., 2009] Weihua Cao, Anastasios A Tsiatis, and Marie Davidian. Improving efficiency and robustness of the doubly robust estimator for a population mean with in- complete data. Biometrika, 96(3):723-734, 2009.","title":"Improving efficiency and robustness of the doubly robust estimator for a population mean with incomplete data","authors":"Weihua Cao, Anastasios A Tsiatis, Marie Davidian","year":"2009","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Carpenter et al., 2006] James R Carpenter, Michael G Ken- ward, and Stijn Vansteelandt. A comparison of multiple imputation and doubly robust estimation for analyses with missing data. Journal of the Royal Statistical Society: Se- ries A (Statistics in Society), 169(3):571-584, 2006.","title":"A comparison of multiple imputation and doubly robust estimation for analyses with missing data","authors":"Michael G James R Carpenter, Stijn Kenward, Vansteelandt","year":"2006","venue":"Journal of the Royal Statistical Society: Series A (Statistics in Society)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chakrabortty and Cai, 2018] Abhishek Chakrabortty and Tianxi Cai. Efficient and adaptive linear regression in semi-supervised settings. The Annals of Statistics, 46(4):1541-1572, 2018.","title":"Efficient and adaptive linear regression in semi-supervised settings","authors":"Cai Chakrabortty, Abhishek Chakrabortty, Tianxi Cai","year":"2018","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chakrabortty et al., 2019] Abhishek Chakrabortty, Jiarui Lu, T Tony Cai, and Hongzhe Li. High dimensional m-estimation with missing outcomes: A semi-parametric framework. arXiv preprint arXiv:1911.11345, 2019.","title":"High dimensional m-estimation with missing outcomes: A semi-parametric framework","authors":"Abhishek Chakrabortty, Jiarui Lu, Tony Cai, Hongzhe Li","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Deng et al., 2020] Siyi Deng, Yang Ning, Jiwei Zhao, and Heping Zhang. Optimal semi-supervised estimation and inference for high-dimensional linear regression. arXiv preprint arXiv:2011.14185, 2020.","title":"Optimal semi-supervised estimation and inference for high-dimensional linear regression","authors":"Siyi Deng, Yang Ning, Jiwei Zhao, Heping Zhang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Farquhar et al., 2021] Sebastian Farquhar, Yarin Gal, and Tom Rainforth. On statistical bias in active learning: How and when to fix it. arXiv preprint arXiv:2101.11665, 2021.","title":"On statistical bias in active learning: How and when to fix it","authors":"Sebastian Farquhar, Yarin Gal, Tom Rainforth","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gal et al., 2017] Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International Conference on Machine Learning, pages 1183-1192. PMLR, 2017.","title":"Deep bayesian active learning with image data","authors":"Yarin Gal, Riashat Islam, Zoubin Ghahramani","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ganti and Gray, 2012] Ravi Ganti and Alexander Gray. Upal: Unbiased pool based active learning. In Artificial Intelligence and Statistics, pages 422-431. PMLR, 2012.","title":"Upal: Unbiased pool based active learning","authors":"Gray ; Ganti, Ravi Ganti, Alexander Gray","year":"2012","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gronsbell and Cai, 2018] Jessica L Gronsbell and Tianxi Cai. Semi-supervised approaches to efficient evaluation of model prediction performance. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 80(3):579-594, 2018.","title":"Semi-supervised approaches to efficient evaluation of model prediction performance","authors":"L Jessica, Tianxi Gronsbell, Cai","year":"2018","venue":"Journal of the Royal Statistical Society: Series B (Statistical Methodology)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gronsbell et al., 2022] Jessica Gronsbell, Molei Liu, Lu Tian, and Tianxi Cai. Efficient evaluation of prediction rules in semi-supervised settings under stratified sam- pling. Journal of the Royal Statistical Society. Series B, Statistical Methodology, 84(4):1353-1391, 2022.","title":"Efficient evaluation of prediction rules in semi-supervised settings under stratified sampling","authors":"Jessica Gronsbell, Molei Liu, Lu Tian, Tianxi Cai","year":"2022","venue":"Journal of the Royal Statistical Society. Series B, Statistical Methodology","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Horvitz and Thompson, 1952] Daniel G Horvitz and Dono- van J Thompson. A generalization of sampling without replacement from a finite universe. Journal of the Ameri- can statistical Association, 47(260):663-685, 1952.","title":"A generalization of sampling without replacement from a finite universe","authors":"G Daniel, Donovan J Horvitz, Thompson","year":"1952","venue":"Journal of the American statistical Association","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hou et al., 2021] Jue Hou, Zijian Guo, and Tianxi Cai. Sur- rogate assisted semi-supervised inference for high dimen- sional risk prediction. arXiv preprint arXiv:2105.01264, 2021.","title":"Surrogate assisted semi-supervised inference for high dimensional risk prediction","authors":"Jue Hou, Zijian Guo, Tianxi Cai","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Imberg et al., 2020] Henrik Imberg, Johan Jonasson, and Marina Axelson-Fisk. Optimal sampling in unbiased ac- tive learning. In International Conference on Artificial In- telligence and Statistics, pages 559-569. PMLR, 2020.","title":"Optimal sampling in unbiased active learning","authors":"Imberg","year":"2020","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Imberg et al., 2022] Henrik Imberg, Xiaomi Yang, Carol Flannagan, and Jonas Bärgman. Active sampling: A machine-learning-assisted framework for finite popula- tion inference with optimal subsamples. arXiv preprint arXiv:2212.10024, 2022.","title":"Active sampling: A machine-learning-assisted framework for finite population inference with optimal subsamples","authors":"Imberg","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kiefer, 1959] Jack Kiefer. Optimum experimental designs. Journal of the Royal Statistical Society: Series B (Method- ological), 21(2):272-304, 1959.","title":"Optimum experimental designs","authors":"Jack Kiefer, Kiefer","year":"1959","venue":"Journal of the Royal Statistical Society: Series B (Methodological)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kossen et al., 2021] Jannik Kossen, Sebastian Farquhar, Yarin Gal, and Tom Rainforth. Active testing: Sample- efficient model evaluation. In International Conference on Machine Learning, pages 5753-5763. PMLR, 2021.","title":"Active testing: Sampleefficient model evaluation","authors":"Jannik Kossen, Sebastian Farquhar, Yarin Gal, Tom Rainforth","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kossen et al., 2022] Jannik Kossen, Sebastian Farquhar, Yarin Gal, and Tom Rainforth. Active surrogate estima- tors: An active learning approach to label-efficient model evaluation. arXiv preprint arXiv:2202.06881, 2022.","title":"Active surrogate estimators: An active learning approach to label-efficient model evaluation","authors":"Jannik Kossen, Sebastian Farquhar, Yarin Gal, Tom Rainforth","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Krijthe and Loog, 2017] Jesse H Krijthe and Marco Loog. Projected estimators for robust semi-supervised classifica- tion. Machine Learning, 106(7):993-1008, 2017.","title":"Projected estimators for robust semi-supervised classification","authors":"H Jesse, Marco Krijthe, Loog","year":"2017","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lee et al., 2021] JooChul Lee, Elizabeth D Schifano, and HaiYing Wang. Fast optimal subsampling probability ap- proximation for generalized linear models. Econometrics and Statistics, 2021.","title":"Fast optimal subsampling probability approximation for generalized linear models","authors":"Lee","year":"2021","venue":"Econometrics and Statistics","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Lee et al., 2022] JooChul Lee, Elizabeth D Schifano, and HaiYing Wang. Sampling-based gaussian mixture regres- sion for big data. Journal of Data Science, pages 1-15, 2022.","title":"Sampling-based gaussian mixture regression for big data","authors":"Lee","year":"2022","venue":"Journal of Data Science","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Liu et al., 2022] Xiaokang Liu, Jessica Chubak, Rebecca A Hubbard, and Yong Chen. Sat: a surrogate-assisted two- wave case boosting sampling method, with application to ehr-based association studies. Journal of the American Medical Informatics Association, 29(5):918-927, 2022.","title":"Sat: a surrogate-assisted twowave case boosting sampling method, with application to ehr-based association studies","authors":"Xiaokang Liu, Jessica Chubak, Rebecca A Hubbard, Yong Chen","year":"2022","venue":"Journal of the American Medical Informatics Association","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[McCullagh, 1983] Peter McCullagh. Quasi-likelihood functions. The Annals of Statistics, 11(1):59-67, 1983.","title":"Quasi-likelihood functions","authors":"Peter Mccullagh","year":"1983","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Robins et al., 1994] James M Robins, Andrea Rotnitzky, and Lue Ping Zhao. Estimation of regression coefficients when some regressors are not always observed. Journal of the American statistical Association, 89(427):846-866, 1994.","title":"Estimation of regression coefficients when some regressors are not always observed","authors":"Andrea James M Robins, Lue Ping Rotnitzky, Zhao","year":"1994","venue":"Journal of the American statistical Association","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Settles et al., 2008] Burr Settles, Mark Craven, and Lewis Friedland. Active learning with real annotation costs. In Proceedings of the NIPS workshop on cost-sensitive learn- ing, volume 1. Vancouver, CA:, 2008.","title":"Active learning with real annotation costs","authors":"Settles","year":"2008","venue":"Proceedings of the NIPS workshop on cost-sensitive learning","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tan and Heagerty, 2020] W Katherine Tan and Patrick J Heagerty. Predictive case control designs for modification learning. arXiv preprint arXiv:2011.14529, 2020.","title":"Predictive case control designs for modification learning","authors":"Katherine Tan, Patrick J Heagerty","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang and Kim, 2022] HaiYing Wang and Jae Kwang Kim. Maximum sampled conditional likelihood for informative subsampling. Journal of Machine Learning Research, 23:1-50, 2022.","title":"Maximum sampled conditional likelihood for informative subsampling","authors":"Wang, Kim ; ] Haiying Wang, Jae Kwang, Kim","year":"2022","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2018] HaiYing Wang, Rong Zhu, and Ping Ma. Optimal subsampling for large sample logistic regres- sion. Journal of the American Statistical Association, 113(522):829-844, 2018.","title":"Optimal subsampling for large sample logistic regression","authors":"Wang","year":"2018","venue":"Journal of the American Statistical Association","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2021] HaiYing Wang, Aonan Zhang, and Chong Wang. Nonuniform negative sampling and log odds correction with rare events data. Advances in Neural In- formation Processing Systems, 34:19847-19859, 2021.","title":"Nonuniform negative sampling and log odds correction with rare events data","authors":"Wang","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wang et al., 2022a] Jing Wang, HaiYing Wang, and Shifeng Xiong. Unweighted estimation based on optimal sam- ple under measurement constraints. Canadian Journal of Statistics, 2022.","title":"Unweighted estimation based on optimal sample under measurement constraints","authors":"Wang","year":"2022","venue":"Canadian Journal of Statistics","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wang et al., 2022b] Linshanshan Wang, Xuan Wang, Katherine P Liao, and Tianxi Cai. Semi-supervised transfer learning for evaluation of model classification performance. arXiv preprint arXiv:2208.07927, 2022.","title":"Semi-supervised transfer learning for evaluation of model classification performance","authors":"Wang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yao and Wang, 2019] Yaqiong Yao and HaiYing Wang. Op- timal subsampling for softmax regression. Statistical Pa- pers, 60(2):585-599, 2019.","title":"Optimal subsampling for softmax regression","authors":"Wang ; Yao, Yaqiong Yao, Haiying Wang","year":"2019","venue":"Statistical Papers","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Yilmaz et al., 2021] Emine Yilmaz, Peter Hayes, Raza Habib, Jordan Burgess, and David Barber. Sample effi- cient model evaluation. arXiv preprint arXiv:2109.12043, 2021.","title":"Sample efficient model evaluation","authors":"Emine Yilmaz, Peter Hayes, Raza Habib, Jordan Burgess, David Barber","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Yoo and Kweon, 2019] Donggeun Yoo and In So Kweon. Learning loss for active learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 93-102, 2019.","title":"Donggeun Yoo and In So Kweon. Learning loss for active learning","authors":"Kweon Yoo","year":"2019","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhang et al., 2021] Tao Zhang, Yang Ning, and David Rup- pert. Optimal sampling for generalized linear models un- der measurement constraints. Journal of Computational and Graphical Statistics, 30(1):106-114, 2021.","title":"Optimal sampling for generalized linear models under measurement constraints","authors":"Tao Zhang, Yang Ning, David Ruppert","year":"2021","venue":"Journal of Computational and Graphical Statistics","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhou et al., 2022] Doudou Zhou, Molei Liu, Mengyan Li, and Tianxi Cai. Doubly robust augmented model accuracy transfer inference with high dimensional features. arXiv preprint arXiv:2208.05134, 2022.","title":"Doubly robust augmented model accuracy transfer inference with high dimensional features","authors":"Doudou Zhou, Molei Liu, Mengyan Li, Tianxi Cai","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhu, 2005] Xiaojin Zhu. Semi-supervised learning with graphs. Carnegie Mellon University, 2005.","title":"Semi-supervised learning with graphs","authors":"Xiaojin Zhu","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}