{"type":"ScholarlyArticle","title":"Generalization Guarantees of Self-Training of Halfspaces under Label Noise Corruption","authors":[{"name":"Lies Hadjadj","slug":"lies-hadjadj","given":"Lies","family":"Hadjadj","affiliation":"Computer Science Laboratory (LIG) , Université Grenoble Alpes , Grenoble , France"},{"name":"Massih-Reza Amini","slug":"massih-reza-amini","given":"Massih-Reza","family":"Amini","affiliation":"Computer Science Laboratory (LIG) , Université Grenoble Alpes , Grenoble , France"},{"name":"Sana Louhichi","slug":"sana-louhichi","given":"Sana","family":"Louhichi","affiliation":"Department of Statistics (LJK) , Université Grenoble Alpes , Grenoble , France"}],"abstract":"We investigate the generalization properties of a self-training algorithm with halfspaces. The approach learns a list of halfspaces iteratively from labeled and unlabeled training data, in which each iteration consists of two steps: exploration and pruning. In the exploration phase, the halfspace is found sequentially by maximizing the unsigned-margin among  unlabeled examples and then assigning pseudo-labels to those that have a distance higher than the current threshold. These pseudo-labels are allegedly corrupted by noise. The training set is then augmented with noisy pseudo-labeled examples, and a new classifier is trained.  This process is repeated until no more unlabeled examples remain for pseudo-labeling. In the pruning phase, pseudo-labeled samples that have a distance to the last halfspace greater than the associated  unsigned-margin are then discarded. We prove that the misclassification error of the resulting sequence of classifiers is bounded and show that the resulting semi-supervised approach never degrades performance compared to the  classifier learned using only the initial labeled training set. Experiments carried out on a variety of benchmarks demonstrate the efficiency of the proposed approach compared to state-of-the-art methods.","doi":"10.24963/ijcai.2023/420","url":"https://ijcai.org/proceedings/2023/420","pdf":"https://ijcai.org/proceedings/2023/0420.pdf","markdown":"https://ijcai.org/proceedings/2023/420.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/420","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":420,"pages":{"first":3777,"last":3785},"keywords":["Machine Learning: ML: Semi-supervised learning","Machine Learning: ML: Learning theory"],"videos":[],"references":[{"position":1,"raw":"[Amini et al., 2009] Massih R. Amini, Nicolas Usunier, and Franc ¸ois Laviolette. A transductive bound for the voted classifier with an application to semi-supervised learning. In Advances in Neural Information Processing Systems, pages 65-72, 2009.","title":"A transductive bound for the voted classifier with an application to semi-supervised learning","authors":"R Massih, Nicolas Amini, Usunier, Franc ¸ois Laviolette","year":"2009","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Balcan and Blum, 2006] Maria-Florina Balcan and Avrim Blum. An augmented PAC model for semi-supervised learning. In Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien, editors, Semi-Supervised Learning, pages 396-419. The MIT Press, 2006.","title":"An augmented PAC model for semi-supervised learning","authors":"Maria-Florina Balcan, Avrim Blum","year":"2006","venue":"Semi-Supervised Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ben-David et al., 2008] Shai Ben-David, Tyler Lu, and Dávid Pál. Does unlabeled data provably help? worst- case analysis of the sample complexity of semi-supervised learning. In Proceedings of the The 21st Annual Confer- ence on Learning Theory (COLT-08), pages 33-44, 2008.","title":"Does unlabeled data provably help? worstcase analysis of the sample complexity of semi-supervised learning","authors":"Shai Ben-David, Tyler Lu, Dávid Pál","year":"2008","venue":"Proceedings of the The 21st Annual Conference on Learning Theory (COLT-08)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bhatia et al., 2015] Kush Bhatia, Himanshu Jain, Purushot- tam Kar, Manik Varma, and Prateek Jain. Sparse local embeddings for extreme multi-label classification. In Ad- vances in Neural Information Processing Systems, vol- ume 28, 2015.","title":"Sparse local embeddings for extreme multi-label classification","authors":"Bhatia","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Boole, 2015] G. Boole. An Investigation of the Laws of Thought: On Which Are Founded the Mathematical Theo- ries of Logic and Probabilities. Creative Media Partners, LLC, 2015.","title":"An Investigation of the Laws of Thought: On Which Are Founded the Mathematical Theories of Logic and Probabilities","authors":"G Boole","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Castelli and Cover, 1995] Vittorio Castelli and Thomas M. Cover. On the exponential value of labeled samples. Pat- tern Recognit. Lett., 16(1):105-111, 1995.","title":"On the exponential value of labeled samples","authors":"Vittorio Castelli, Thomas M Cover","year":"1995","venue":"Pattern Recognit. Lett","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chapelle et al., 2006] Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien, editors. Semi-Supervised Learning. The MIT Press, 2006.","title":"Semi-Supervised Learning","authors":null,"year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chawla and Karakoulas, 2011] Nitesh Chawla and Grigoris Karakoulas. Learning from labeled and unlabeled data: An empirical study across techniques and domains. Journal of Artificial Intelligence Research -JAIR, 23, 09 2011.","title":"Learning from labeled and unlabeled data: An empirical study across techniques and domains","authors":"Nitesh Chawla, Grigoris Karakoulas","year":"2011","venue":"Journal of Artificial Intelligence Research -JAIR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Cortes and Vapnik, 1995] Corinna Cortes and Vladimir Vapnik. Support-vector networks. Mach. Learn., 20(3):273-297, September 1995.","title":"Support-vector networks","authors":"Corinna Cortes, Vladimir Vapnik","year":"1995","venue":"Mach. Learn","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Diakonikolas and Kane, 2020] Ilias Diakonikolas and Daniel M. Kane. Hardness of learning halfspaces with massart noise. CoRR, abs/2012.09720, 2020.","title":"Hardness of learning halfspaces with massart noise","authors":"Ilias Diakonikolas, Daniel M Kane","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Diakonikolas et al., 2019] Ilias Diakonikolas, Themis Gouleakis, and Christos Tzamos. Distribution- independent pac learning of halfspaces with massart noise. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32, pages 4749-4760.","title":"Distributionindependent pac learning of halfspaces with massart noise","authors":"Ilias Diakonikolas, Themis Gouleakis, Christos Tzamos","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"Curran Associates, Inc., 2019.","title":null,"authors":"Curran Associates, Inc","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Diakonikolas et al., 2021] Ilias Diakonikolas, Russell Im- pagliazzo, Daniel M. Kane, Rex Lei, Jessica Sorrell, and Christos Tzamos. Boosting in the presence of massart noise. In Mikhail Belkin and Samory Kpotufe, editors, Proceedings of Thirty Fourth Conference on Learning Theory, volume 134 of Proceedings of Machine Learning Research, pages 1585-1644. PMLR, 15-19 Aug 2021.","title":"Boosting in the presence of massart noise","authors":"Ilias Diakonikolas, Russell Impagliazzo, Daniel M Kane, Rex Lei, Jessica Sorrell, Christos Tzamos","year":"2021","venue":"Proceedings of Thirty Fourth Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Dua and Graff, 2019] Dheeru Dua and Casey Graff. UCI machine learning repository. https://archive.ics.uci.edu/ ml, 2019. Accessed: 2023-06-27.","title":"Dheeru Dua and Casey Graff. UCI machine learning repository","authors":"Graff Dua","year":"2019","venue":null,"doi":null,"url":"https://archive.ics.uci.edu/ml,2019","internal_url":null},{"position":15,"raw":"[Duchi, 2016] John C. Duchi. Introductory lectures on stochastic convex optimization. Park City Mathematics Series, Graduate Summer School Lectures, 2016.","title":"Introductory lectures on stochastic convex optimization","authors":"John C Duchi","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Dvoretzky et al., 1956] A. Dvoretzky, J. Kiefer, and J. Wol- fowitz. Asymptotic Minimax Character of the Sample Dis- tribution Function and of the Classical Multinomial Esti- mator. The Annals of Mathematical Statistics, 27(3):642 - 669, 1956.","title":"Asymptotic Minimax Character of the Sample Distribution Function and of the Classical Multinomial Estimator","authors":"A Dvoretzky, J Kiefer, J Wolfowitz","year":"1956","venue":"The Annals of Mathematical Statistics","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Fan et al., 2008] Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Jen Lin. Liblinear: A library for large linear classification. J. Mach. Learn. Res., 9:1871-1874, June 2008.","title":"Liblinear: A library for large linear classification","authors":"Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, Chih-Jen Lin","year":"2008","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Frei et al., 2021] Spencer Frei, Difan Zou, Zixiang Chen, and Quanquan Gu. Self-training converts weak learners to strong learners in mixture models. CoRR, abs/2106.13805, 2021.","title":"Self-training converts weak learners to strong learners in mixture models","authors":"Spencer Frei, Difan Zou, Zixiang Chen, Quanquan Gu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Grandvalet and Bengio, 2005] Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimiza- tion. In L. Saul, Y. Weiss, and L. Bottou, editors, Ad- vances in Neural Information Processing Systems, vol- ume 17, pages 529-536. MIT Press, 2005.","title":"Semi-supervised learning by entropy minimization","authors":"Bengio Grandvalet, Yves Grandvalet, Yoshua Bengio","year":"2005","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Joachims, 1999] Thorsten Joachims. Transductive inference for text classification using support vector machines. In Proceedings of the Sixteenth International Conference on Machine Learning, ICML '99, page 200-209, San Fran- cisco, CA, USA, 1999.","title":"Transductive inference for text classification using support vector machines","authors":"Thorsten Joachims","year":"1999","venue":"Proceedings of the Sixteenth International Conference on Machine Learning, ICML '99","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Johnson et al., 2020] David S. Johnson, Lee Breslau, Ilias Diakonikolas, Nick Duffield, Yu Gu, MohammadTaghi Hajiaghayi, Howard J. Karloff, Mauricio G. C. Resende, and Subhabrata Sen. Near-optimal disjoint-path facility location through set cover by pairs. Oper. Res., 68(3):896- 926, 2020.","title":"Near-optimal disjoint-path facility location through set cover by pairs","authors":"David S Johnson, Lee Breslau, Ilias Diakonikolas, Nick Duffield, Yu Gu, Mohammadtaghi Hajiaghayi, Howard J Karloff, G C Mauricio, Subhabrata Resende, Sen","year":"2020","venue":"Oper. Res","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Krijthe, 2017] Jesse H. Krijthe. Rssl: Semi-supervised learning in r. In Bertrand Kerautret, Miguel Colom, and Pascal Monasse, editors, Reproducible Research in Pat- tern Recognition, pages 104-115, Cham, 2017. Springer International Publishing.","title":"Rssl: Semi-supervised learning in r","authors":"Jesse H Krijthe","year":"2017","venue":"Reproducible Research in Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Li and Zhou, 2011] Yu-Feng Li and Zhi-Hua Zhou. To- wards Making Unlabeled Data Never Hurt. In Proceedings of the 28th International Conference on Machine Learn- ing, pages 1081-1088, 2011.","title":"Towards Making Unlabeled Data Never Hurt","authors":"Yu-Feng Li, Zhi-Hua Zhou","year":"2011","venue":"Proceedings of the 28th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Massart and Nédélec, 2006] Pascal Massart and Elodie Nédélec. Risk bounds for statistical learning. Ann. Statist., 34(5):2326-2366, 10 2006.","title":"Risk bounds for statistical learning","authors":"Pascal Massart, Elodie Nédélec","year":"2006","venue":"Ann. Statist","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Maximov et al., 2018] Yury Maximov, Massih-Reza Amini, and Zaïd Harchaoui. Rademacher complexity bounds for a penalized multi-class semi-supervised algorithm. Journal of Artificial Intelligence Research, 61:761-786, 2018.","title":"Rademacher complexity bounds for a penalized multi-class semi-supervised algorithm","authors":"Yury Maximov, Massih-Reza Amini, Zaïd Harchaoui","year":"2018","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Montasser et al., 2020] Omar Montasser, Surbhi Goel, Ilias Diakonikolas, and Nathan Srebro. Efficiently learning ad- versarially robust halfspaces with noise. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th Interna- tional Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 7010- 7021. PMLR, 13-18 Jul 2020.","title":"Efficiently learning adversarially robust halfspaces with noise","authors":"Omar Montasser, Surbhi Goel, Ilias Diakonikolas, Nathan Srebro","year":"2020","venue":"Proceedings of the 37th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Niyogi, 2013] Partha Niyogi. Manifold regularization and semi-supervised learning: Some theoretical analyses. Journal of Machine Learning Research, 14(1):1229-1250, 2013.","title":"Manifold regularization and semi-supervised learning: Some theoretical analyses","authors":"Partha Niyogi","year":"2013","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Rigollet, 2007] Philippe Rigollet. Generalization error bounds in semi-supervised classification under the clus- ter assumption. Journal of Machine Learning Research, 8:1369-1392, 2007.","title":"Generalization error bounds in semi-supervised classification under the cluster assumption","authors":"Philippe Rigollet","year":"2007","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Singh et al., 2009] Aarti Singh, Robert Nowak, and Jerry Zhu. Unlabeled data: Now it helps, now it doesn't. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, edi- tors, Advances in Neural Information Processing Systems, volume 21.","title":"Unlabeled data: Now it helps, now it doesn't","authors":"Aarti Singh, Robert Nowak, Jerry Zhu","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"Curran Associates, Inc., 2009.","title":null,"authors":"Curran Associates, Inc","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wei et al., 2020] Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical analysis of self-training with deep networks on unlabeled data. CoRR, abs/2010.03622, 2020.","title":"Theoretical analysis of self-training with deep networks on unlabeled data","authors":"Colin Wei, Kendrick Shen, Yining Chen, Tengyu Ma","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wei et al., 2021] Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical analysis of self-training with deep networks on unlabeled data. In 9th International Conference on Learning Representations, ICLR 2021, Vir- tual Event, Austria, May 3-7, 2021, 2021.","title":"Theoretical analysis of self-training with deep networks on unlabeled data","authors":"Colin Wei, Kendrick Shen, Yining Chen, Tengyu Ma","year":"2021","venue":"9th International Conference on Learning Representations, ICLR 2021, Virtual Event","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wolfe, 2012] Douglas A. Wolfe. Nonparametrics: Statisti- cal Methods Based on Ranks and Its Impact on the Field of Nonparametric Statistics, pages 1101-1110. Springer US, Boston, MA, 2012.","title":"Nonparametrics: Statistical Methods Based on Ranks and Its Impact on the Field of Nonparametric Statistics","authors":"Douglas A Wolfe","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhu and Ghahramani, 2002] Xiaojin Zhu and Zoubin Ghahramani. Learning from labeled and unlabeled data with label propagation. Technical report, CMU-CALD- 02-107, Carnegie Mellon University, 2002.","title":"Learning from labeled and unlabeled data with label propagation","authors":"Xiaojin Zhu, Zoubin Ghahramani","year":"2002","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhu, 2005] Xiaojin Zhu. Semi-supervised learning litera- ture survey. Technical Report 1530, Computer Sciences, University of Wisconsin-Madison, 2005.","title":"Semi-supervised learning literature survey","authors":"Xiaojin Zhu","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}