{"type":"ScholarlyArticle","title":"VCC-INFUSE: Towards Accurate and Efficient Selection of Unlabeled Examples in Semi-supervised Learning","authors":[{"name":"Shijie Fang","slug":"shijie-fang","given":"Shijie","family":"Fang","affiliation":"National Key Laboratory of General Artificial Intelligence , School of Intelligence Science and Technology , Peking University 2 Google , Shanghai , China"},{"name":"Qianhan Feng","slug":"qianhan-feng","given":"Qianhan","family":"Feng","affiliation":"National Key Laboratory of General Artificial Intelligence , School of Intelligence Science and Technology , Peking University 2 Google , Shanghai , China"},{"name":"Tong Lin","slug":"tong-lin","given":"Tong","family":"Lin","affiliation":"National Key Laboratory of General Artificial Intelligence , School of Intelligence Science and Technology , Peking University 2 Google , Shanghai , China"}],"abstract":"Despite the progress of Semi-supervised Learning (SSL), existing methods fail to utilize unlabeled data effectively and efficiently. Many pseudo-label-based methods select unlabeled examples based on inaccurate confidence scores from the classifier. Most prior work also uses all available unlabeled data without pruning, making it difficult to handle large amounts of unlabeled data. To address these issues, we propose two methods: Variational Confidence Calibration (VCC) and Influence-Function-based Unlabeled Sample Elimination (INFUSE). VCC is a universal plugin for SSL confidence calibration, using a variational autoencoder to select more accurate pseudo labels based on three types of consistency scores. INFUSE is a data pruning method that constructs a core dataset of unlabeled examples under SSL. Our methods are effective in multiple datasets and settings, reducing classification error rates and saving training time. Together, VCC-INFUSE reduces the error rate of FlexMatch on the CIFAR-100 dataset by 1.08% while saving nearly half of the training time.","doi":"10.24963/ijcai.2024/437","url":"https://ijcai.org/proceedings/2024/437","pdf":"https://ijcai.org/proceedings/2024/0437.pdf","markdown":"https://ijcai.org/proceedings/2024/437.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/437","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":437,"pages":{"first":3953,"last":3961},"keywords":["Machine Learning: ML: Semi-supervised learning"],"videos":[],"references":[{"position":1,"raw":"Method 400 labels 2500 labels 10000 labels ER","title":"Method","authors":null,"year":null,"venue":"Method","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"ECE MCE ACE ER(%) ECE MCE ACE ER(%) ECE MCE ACE FixMatch 46.42 0.382 0.573 0.376 28.03 0.208 0.530 0.199 22.20 0.127 0.322 0.128 VCC-FixMatch 43.29 0.359 0.560 0.345 27.81 0.195 0.418 0.182 22.01 0.125 0.317 0.127 FlexMatch 39.94 0.291 0.512 0.286 26.49 0.169 0.369 0.173 21.90 0.120 0.311 0.126 VCC-FlexMatch 37.52 0.257 0.446 0.258 25.26 0.147 0.324 0.163 21.55 0.104 0.269 0.125 SimMatch 37.81 0.325 0.510 0.328 25.07 0.157 0.358 0.179 20.58 0.113 0.295 0.116 VCC-SimMatch 37.20 0.317 0.514 0.314 25.01 0.155 0.347 0.173 20.61 0.115 0.291 0.121","title":"ECE MCE ACE ER","authors":"Ece Mce, Ace Er","year":null,"venue":"ECE MCE ACE FixMatch","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Berthelot et al., 2020] David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. ReMixMatch: Semi-Supervised Learn- ing with Distribution Matching and Augmentation An- choring. In International Conference on Learning Rep- resentations, Addis Ababa, Ethiopia, 2020.","title":"ReMixMatch: Semi-Supervised Learning with Distribution Matching and Augmentation Anchoring","authors":"David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, Colin Raffel","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2023] Hao Chen, Ran Tao, Yue Fan, Yidong Wang, Jindong Wang, Bernt Schiele, Xing Xie, Bhiksha Raj, and Marios Savvides. Softmatch: Addressing the quantity-quality tradeoff in semi-supervised learning. In The Eleventh International Conference on Learning Rep- resentations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, 2023.","title":"Softmatch: Addressing the quantity-quality tradeoff in semi-supervised learning","authors":"Ran Hao Chen, Yue Tao, Yidong Fan, Jindong Wang, Bernt Wang, Xing Schiele, Bhiksha Xie, Marios Raj, Savvides","year":"2023","venue":"The Eleventh International Conference on Learning Representations, ICLR 2023","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Coates et al., 2011] Adam Coates, Andrew Y. Ng, and Honglak Lee. An Analysis of Single-Layer Networks in Unsupervised Feature Learning. In Proceedings of the Fourteenth International Conference on Artificial Intelli- gence and Statistics, Fort Lauderdale, USA, 2011.","title":"An Analysis of Single-Layer Networks in Unsupervised Feature Learning","authors":"Adam Coates, Andrew Y Ng, Honglak Lee","year":"2011","venue":"Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In Conference on Computer Vision and Pattern Recognition, Miami, Florida, USA, 2009.","title":"ImageNet: A Large-Scale Hierarchical Image Database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Feng et al., 2024] Qianhan Feng, Lujing Xie, Shijie Fang, and Tong Lin. Bacon: Boosting imbalanced semi- supervised learning via balanced feature-level contrastive learning. In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, 2024.","title":"Bacon: Boosting imbalanced semisupervised learning via balanced feature-level contrastive learning","authors":"Feng","year":"2024","venue":"Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gal and Ghahramani, 2016] Yarin Gal and Zoubin Ghahra- mani. Dropout as a Bayesian Approximation: Represent- ing Model Uncertainty in Deep Learning. In Proceedings of the 33nd International Conference on Machine Learn- ing, New York City, NY, USA, 2016.","title":"Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning","authors":"Ghahramani Gal","year":"2016","venue":"Proceedings of the 33nd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ganev and Aitchison, 2020] Stoil Ganev and Laurence Aitchison. Semi-supervised Learning Objectives as Log- likelihoods in a Generative Model of Data Curation. arXiv preprint arXiv:2008.05913, 2020.","title":"Semi-supervised Learning Objectives as Loglikelihoods in a Generative Model of Data Curation","authors":"Aitchison Ganev, Stoil Ganev, Laurence Aitchison","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Guo et al., 2017] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On Calibration of Modern Neural Networks. In International Conference on Machine Learn- ing, Sydney, NSW, Australia, 2017.","title":"On Calibration of Modern Neural Networks","authors":"Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q Weinberger","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Killamsetty et al., 2021a] KrishnaTeja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, Abir De, and Rishabh K. Iyer. GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training. In Proceedings of the International Conference on Machine Learning, Virtual Event, 2021.","title":"GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training","authors":"Killamsetty","year":"2021","venue":"Proceedings of the International Conference on Machine Learning, Virtual Event","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Killamsetty et al., 2021b] KrishnaTeja Killamsetty, Xu- jiang Zhao, Feng Chen, and Rishabh K. Iyer. RETRIEVE: Coreset Selection for Efficient and Robust Semi- Supervised Learning. In Advances in Neural Information Processing Systems, virtual, 2021.","title":"RETRIEVE: Coreset Selection for Efficient and Robust Semi-Supervised Learning","authors":"Killamsetty","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kingma and Welling, 2014] Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In Inter- national Conference on Learning Representations, Banff, AB, Canada, 2014.","title":"Auto-Encoding Variational Bayes","authors":"P Diederik, Max Kingma, Welling","year":"2014","venue":"Inter-national Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Koh and Liang, 2017] Pang Wei Koh and Percy Liang. Un- derstanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, 2017.","title":"Understanding black-box predictions via influence functions","authors":"Pang Wei, Koh, Percy Liang","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning, ICML 2017","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Krizhevsky et al., 2009] Alex Krizhevsky, Geoffrey Hinton, et al. Learning Multiple Layers of Features from Tiny Images. In Doctoral dissertation, University of Toronto, 2009.","title":"Learning Multiple Layers of Features from Tiny Images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kumar et al., 2018] Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable Calibration Measures For Neural Networks From Kernel Mean Embeddings. In Interna- tional Conference on Machine Learning, Stockholm, Swe- den, 2018.","title":"Trainable Calibration Measures For Neural Networks From Kernel Mean Embeddings","authors":"Aviral Kumar, Sunita Sarawagi, Ujjwal Jain","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Laine and Aila, 2016] Samuli Laine and Timo Aila. Tem- poral Ensembling for Semi-Supervised Learning. arXiv preprint arXiv:1610.02242, 2016.","title":"Temporal Ensembling for Semi-Supervised Learning","authors":"Laine, Aila ;, Samuli Laine, Timo Aila","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lee et al., 2021] Hyuck Lee, Seungjae Shin, and Heey- oung Kim. ABC: Auxiliary balanced classifier for class- imbalanced semi-supervised learning. In Advances in Neu- ral Information Processing Systems, 2021.","title":"ABC: Auxiliary balanced classifier for classimbalanced semi-supervised learning","authors":"Hyuck Lee, Seungjae Shin, Heeyoung Kim","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li et al., 2021] Junnan Li, Caiming Xiong, and Steven C. H. Hoi. Comatch: Semi-supervised learning with contrastive graph regularization. In IEEE/CVF International Confer- ence on Computer Vision, Montreal, QC, Canada, October 10-17, 2021.","title":"Comatch: Semi-supervised learning with contrastive graph regularization","authors":"Li","year":"2021","venue":"IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge J. Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Com- mon Objects in Context. In European Conference on Com- puter Vision, Zurich, Switzerland, 2014.","title":"Microsoft COCO: Common Objects in Context","authors":"Tsung-Yi Lin, Michael Maire, Serge J Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence Zitnick","year":"2014","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Luketina et al., 2016] Jelena Luketina, Tapani Raiko, Math- ias Berglund, and Klaus Greff. Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters. In International Conference on Machine Learning, New York City, NY, USA, 2016.","title":"Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters","authors":"Luketina","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Miyato et al., 2019] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(8):1979-1993, 2019.","title":"Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning","authors":"Miyato","year":"2019","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Netzer et al., 2011] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Read- ing Digits in Natural Images with Unsupervised Feature Learning. In NIPS Workshop on Deep Learning and Un- supervised Feature Learning, 2011.","title":"Reading Digits in Natural Images with Unsupervised Feature Learning","authors":"Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, Andrew Y Ng","year":"2011","venue":"NIPS Workshop on Deep Learning and Unsupervised Feature Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Nixon et al., 2019] Jeremy Nixon, Michael W. Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Mea- suring Calibration in Deep Learning. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, Long Beach, CA, USA, 2019.","title":"Measuring Calibration in Deep Learning","authors":"Jeremy Nixon, Michael W Dusenberry, Linchuan Zhang, Ghassen Jerfel, Dustin Tran","year":"2019","venue":"IEEE Conference on Computer Vision and Pattern Recognition Workshops","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Paul et al., 2021] Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep Learning on a Data Diet: Finding Important Examples Early in Training. In Advances in Neural Information Processing Systems, vir- tual, 2021.","title":"Deep Learning on a Data Diet: Finding Important Examples Early in Training","authors":"Paul","year":"2021","venue":"Advances in Neural Information Processing Systems, virtual","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sohn et al., 2020] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin Raffel, Ekin Do- gus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fix- Match: Simplifying Semi-Supervised Learning with Con- sistency and Confidence. In Advances in Neural Informa- tion Processing Systems, Virtual, 2020.","title":"Fix-Match: Simplifying Semi-Supervised Learning with Consistency and Confidence","authors":"Sohn","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xu et al., 2015] Chang Xu, Dacheng Tao, and Chao Xu. Multi-View Learning With Incomplete Views. IEEE Trans. Image Process., 24(12):5812-5825, 2015.","title":"Multi-View Learning With Incomplete Views","authors":"Chang Xu, Dacheng Tao, Chao Xu","year":"2015","venue":"IEEE Trans. Image Process","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2020] Jize Zhang, Bhavya Kailkhura, and Thomas Yong-Jin Han. Mix-N-Match : Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learning. In International Conference on Machine Learning, Virtual, 2020.","title":"Mix-N-Match : Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learning","authors":"Zhang","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2021] Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. FlexMatch: Boosting Semi- Supervised Learning with Curriculum Pseudo Labeling. In Advances in Neural Information Processing, Virtual, 2021.","title":"FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling","authors":"Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, Takahiro Shinozaki","year":"2021","venue":"Advances in Neural Information Processing","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zheng et al., 2022] Mingkai Zheng, Shan You, Lang Huang, Fei Wang, Chen Qian, and Chang Xu. SimMatch: Semi-supervised Learning with Similarity Matching. In Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, 2022.","title":"SimMatch: Semi-supervised Learning with Similarity Matching","authors":"Zheng","year":"2022","venue":"Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhou et al., 2020] Tianyi Zhou, Shengjie Wang, and Jeff A. Bilmes. Time-Consistent Self-Supervision for Semi- Supervised Learning. In International Conference on Ma- chine Learning, Virtual, 2020.","title":"Time-Consistent Self-Supervision for Semi-Supervised Learning","authors":"Tianyi Zhou, Shengjie Wang, Jeff A Bilmes","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}