{"type":"ScholarlyArticle","title":"SELC: Self-Ensemble Label Correction Improves Learning with Noisy Labels","authors":[{"name":"Yangdi Lu","slug":"yangdi-lu","given":"Yangdi","family":"Lu","affiliation":"Department of Computing and Software , McMaster University , Canada"},{"name":"Wenbo He","slug":"wenbo-he","given":"Wenbo","family":"He","affiliation":"Department of Computing and Software , McMaster University , Canada"}],"abstract":"Deep neural networks are prone to overfitting noisy labels, resulting in poor generalization performance. To overcome this problem, we present a simple and effective method self-ensemble label correction (SELC) to progressively correct noisy labels and refine the model. We look deeper into the memorization behavior in training with noisy labels and observe that the network outputs are reliable in the early stage. To retain this reliable knowledge, SELC uses ensemble predictions formed by an exponential moving average of network outputs to update the original noisy labels. We show that training with SELC refines the model by gradually reducing supervision from noisy labels and increasing supervision from ensemble predictions. Despite its simplicity, compared with many state-of-the-art methods, SELC obtains more promising and stable results in the presence of class-conditional, instance-dependent, and real-world label noise. The code is available at https://github.com/MacLLL/SELC.","doi":"10.24963/ijcai.2022/455","url":"https://ijcai.org/proceedings/2022/455","pdf":"https://ijcai.org/proceedings/2022/0455.pdf","markdown":"https://ijcai.org/proceedings/2022/455.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/455","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":455,"pages":{"first":3278,"last":3284},"keywords":["Machine Learning: Weakly Supervised Learning","Machine Learning: Ensemble Methods","Machine Learning: Classification"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986575","length_sec":85,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984718","length_sec":318,"provider":"slideslive"}],"references":[{"position":1,"raw":"Dataset Noise Cross Entropy Co-teaching+ GCE SCE LRT PLC SELC (ours) [","title":"Dataset Noise Cross Entropy Co-teaching+ GCE SCE LRT PLC SELC","authors":null,"year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Yu et al., 2019] [","title":null,"authors":"Yu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Zhang and Sabuncu, 2018] [","title":null,"authors":"Sabuncu Zhang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Wang et al., 2019] [","title":null,"authors":"Wang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Zheng et al., 2020] [","title":null,"authors":"Zheng","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Zhang et al., 2020] CIFAR-10 Type-I (35%) 78.11 ± 0.74 79.97 ± 0.15 80.65 ± 0.39 79.76 ± 0.72 80.98 ± 0.80 82.80 ± 0.27 86.97 ± 0.15 Type-I (70%) 41.98 ± 1.96 40.69 ± 1.99 36.52 ± 1.62 36.29 ± 0.66 41.52 ± 4.53 42.74 ± 2.14 43.78 ± 2.64 Type-II (35%) 76.65 ± 0.57 77.34 ± 0.44 77.60 ± 0.88 77.92 ± 0.89 80.74 ± 0.25 81.54 ± 0.47 87.06 ± 0.20 Type-II (70%) 45.57 ± 1.12 45.44 ± 0.64 40.30 ± 1.46 41.11 ± 1.92 44.67 ± 3.89 46.04 ± 2.20 46.79 ± 3.06 Type-III (35%) 76.89 ± 0.79 78.38 ± 0.67 79.18 ± 0.61 78.81 ± 0.29 81.08 ± 0.35 81.50 ± 0.50 87.31 ± 0.18 Type-III (70%) 43.32 ± 1.00 41.90 ± 0.86 37.10 ± 0.59 38.49 ± 1.46 44.47 ± 1.23 45.05 ± 1.13 45.57 ± 1.71 CIFAR-100 Type-I (35%) 57.68 ± 0.29 56.70 ± 0.71 58.37 ± 0.18 55.20 ± 0.33 56.74 ± 0.34 60.01 ± 0.43 65.72 ± 0.17 Type-I (70%) 39.32 ± 0.43 39.53 ± 0.28 40.01 ± 0.71 40.02 ± 0.85 45.29 ± 0.43 45.92 ± 0.61 49.72 ± 0.15 Type-II (35%) 57.83 ± 0.25 56.57 ± 0.52 58.11 ± 1.05 56.10 ± 0.73 57.25 ± 0.68 63.68 ± 0.29 66.79 ± 0.18 Type-II (70%) 39.30 ± 0.32 36.84 ± 0.39 37.75 ± 0.46 38.45 ± 0.45 43.71 ± 0.51 45.03 ± 0.50 52.65 ± 0.26 Type-III (35%) 56.07 ± 0.79 55.77 ± 0.98 57.51 ± 1.16 56.04 ± 0.74 56.57 ± 0.30 63.68 ± 0.29 66.41 ± 0.17 Type-III (70%) 40.01 ± 0.18 35.37 ± 2.65 40.53 ± 0.60 39.94 ± 0.84 44.41 ± 0.19 44.45 ± 0.62 49.85","title":"CIFAR-10 Type-I","authors":"Zhang","year":"2020","venue":"CIFAR-10 Type-I","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Arazo et al., 2019] Eric Arazo, Diego Ortego, Paul Albert, Noel O'Connor, and Kevin Mcguinness. Unsupervised label noise modeling and loss correction. In ICML, pages 312-321, 2019.","title":"Unsupervised label noise modeling and loss correction","authors":"Eric Arazo, Diego Ortego, Paul Albert, O' Noel, Kevin Connor, Mcguinness","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Arpit et al., 2017] Devansh Arpit, Stanislaw K Jastrzebski, Nico- las Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kan- wal, Tegan Maharaj, Asja Fischer, Aaron C Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In ICML, 2017.","title":"A closer look at memorization in deep networks","authors":"Devansh Arpit, Stanislaw K Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, S Maxinder, Tegan Kanwal, Asja Maharaj, Aaron C Fischer, Yoshua Courville, Bengio","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chen et al., 2019] Pengfei Chen, Ben Ben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. In ICML, pages 1062-1070, 2019.","title":"Understanding and utilizing deep neural networks trained with noisy labels","authors":"Pengfei Chen, Ben Ben Liao, Guangyong Chen, Shengyu Zhang","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Chen et al., 2021a] Pengfei Chen, Junjie Ye, Guangyong Chen, Jingwei Zhao, and Pheng-Ann Heng. Beyond class-conditional assumption: A primary attempt to combat instance-dependent la- bel noise. In AAAI, volume 35, pages 11442-11450, 2021.","title":"Beyond class-conditional assumption: A primary attempt to combat instance-dependent label noise","authors":"Chen","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Chen et al., 2021b] Yingyi Chen, Xi Shen, Shell Xu Hu, and Jo- han AK Suykens. Boosting co-teaching with compression regu- larization for label noise. In CVPR, pages 2688-2692, 2021.","title":"Boosting co-teaching with compression regularization for label noise","authors":"Chen","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gui et al., 2021] Xian-Jin Gui, Wei Wang, and Zhang-Hao Tian. Towards understanding deep learning from noisy labels with small-loss criterion. IJCAI, 2021.","title":"Towards understanding deep learning from noisy labels with small-loss criterion","authors":"Xian-Jin Gui, Wei Wang, Zhang-Hao Tian","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/340"},{"position":13,"raw":"[Han et al., 2018] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co- teaching: Robust training of deep neural networks with extremely noisy labels. In NIPS, pages 8527-8537, 2018.","title":"Coteaching: Robust training of deep neural networks with extremely noisy labels","authors":"Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, Masashi Sugiyama","year":"2018","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kim et al., 2019] Youngdong Kim, Junho Yim, Juseung Yun, and Junmo Kim. Nlnl: Negative learning for noisy labels. In ICCV, pages 101-110, 2019.","title":"Nlnl: Negative learning for noisy labels","authors":"Kim","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Krizhevsky et al., 2009] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2017] Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. Webvision database: Visual learning and understanding from web data. arXiv, 2017.","title":"Webvision database: Visual learning and understanding from web data","authors":"Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, Luc Van Gool","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2020] Sheng Liu, Jonathan Niles-Weed, Narges Raza- vian, and Carlos Fernandez-Granda. Early-learning regulariza- tion prevents memorization of noisy labels. NIPS, 33, 2020.","title":"Early-learning regularization prevents memorization of noisy labels","authors":"Sheng Liu, Jonathan Niles-Weed, Narges Razavian, Carlos Fernandez-Granda","year":"2020","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lu et al., 2021] Yangdi Lu, Yang Bo, and Wenbo He. Confidence adaptive regularization for deep learning with noisy labels. arXiv preprint arXiv:2108.08212, 2021.","title":"Confidence adaptive regularization for deep learning with noisy labels","authors":"Lu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lu et al., 2022] Yangdi Lu, Yang Bo, and Wenbo He. An ensemble model for combating label noise. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, pages 608-617, 2022.","title":"An ensemble model for combating label noise","authors":"Lu","year":"2022","venue":"Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ma et al., 2020] Xingjun Ma, Hanxun Huang, Yisen Wang, Si- mone Romano, Sarah Erfani, and James Bailey. Normalized loss functions for deep learning with noisy labels. In In ICML, pages 6543-6553. PMLR, 2020.","title":"Normalized loss functions for deep learning with noisy labels","authors":"Ma","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Mirzasoleiman et al., 2020] Baharan Mirzasoleiman, Kaidi Cao, and Jure Leskovec. Coresets for robust training of neural net- works against noisy labels. NIPS, 2020.","title":"Coresets for robust training of neural networks against noisy labels","authors":"Mirzasoleiman","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Natarajan et al., 2013] Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. NIPS, 26:1196-1204, 2013.","title":"Learning with noisy labels","authors":"Nagarajan Natarajan, Pradeep K Inderjit S Dhillon, Ambuj Ravikumar, Tewari","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Nguyen et al., 2020] Tam Nguyen, C Mummadi, T Ngo, L Beggel, and Thomas Brox. Self: learning to filter noisy labels with self- ensembling. In ICLR, 2020.","title":"Self: learning to filter noisy labels with selfensembling","authors":"Tam Nguyen, C Mummadi, T Ngo, L Beggel, Thomas Brox","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Patrini et al., 2017] Giorgio Patrini, Alessandro Rozza, Aditya Kr- ishna Menon, Richard Nock, and Lizhen Qu. Making deep neu- ral networks robust to label noise: A loss correction approach. In CVPR, pages 1944-1952, 2017.","title":"Making deep neural networks robust to label noise: A loss correction approach","authors":"Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, Lizhen Qu","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Permuter et al., 2006] Haim Permuter, Joseph Francos, and Ian Jermyn. A study of gaussian mixture models of color and tex- ture features for image classification and segmentation. Pattern recognition, 39(4):695-706, 2006.","title":"Haim Permuter, Joseph Francos, and Ian Jermyn. A study of gaussian mixture models of color and texture features for image classification and segmentation","authors":"Permuter","year":"2006","venue":"Pattern recognition","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Reed et al., 2015] Scott E Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. In ICLR (Workshop), 2015.","title":"Training deep neural networks on noisy labels with bootstrapping","authors":"Honglak Scott E Reed, Dragomir Lee, Christian Anguelov, Dumitru Szegedy, Andrew Erhan, Rabinovich","year":"2015","venue":"ICLR (Workshop)","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Song et al., 2019] Hwanjun Song, Minseok Kim, and Jae-Gil Lee. Selfie: Refurbishing unclean samples for robust deep learning. In ICML, pages 5907-5915, 2019.","title":"Selfie: Refurbishing unclean samples for robust deep learning","authors":"Song","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Tanaka et al., 2018] Daiki Tanaka, Daiki Ikami, Toshihiko Ya- masaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. In CVPR, pages 5552-5560, 2018.","title":"Joint optimization framework for learning with noisy labels","authors":"Tanaka","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Thulasidasan et al., 2019] Sunil Thulasidasan, Tanmoy Bhat- tacharya, Jeff Bilmes, Gopinath Chennupati, and Jamal Mohd- Yusof. Combating label noise in deep learning using abstention. In ICML, pages 6234-6243. PMLR, 2019.","title":"Combating label noise in deep learning using abstention","authors":"Tanmoy Sunil Thulasidasan, Jeff Bhattacharya, Gopinath Bilmes, Jamal Chennupati, Mohd-Yusof","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wang et al., 2019] Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In ICCV, pages 322-330, 2019.","title":"Symmetric cross entropy for robust learning with noisy labels","authors":"Wang","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Xiao et al., 2015] Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In CVPR, pages 2691-2699, 2015.","title":"Learning from massive noisy labeled data for image classification","authors":"Tong Xiao, Tian Xia, Yi Yang, Chang Huang, Xiaogang Wang","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Xu et al., 2019] Yilun Xu, Peng Cao, Yuqing Kong, and Yizhou Wang. L dmi: A novel information-theoretic loss function for training deep nets robust to label noise. Advances in Neural In- formation Processing Systems, 32:6225-6236, 2019.","title":"L dmi: A novel information-theoretic loss function for training deep nets robust to label noise","authors":"Yilun Xu, Peng Cao, Yuqing Kong, Yizhou Wang","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Yi and Wu, 2019] Kun Yi and Jianxin Wu. Probabilistic end-to- end noise correction for learning with noisy labels. In CVPR, pages 7017-7025, 2019.","title":"Probabilistic end-toend noise correction for learning with noisy labels","authors":"Yi, Wu ; ] Kun Yi, Jianxin Wu","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Yu et al., 2019] Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, IW Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In ICML, 2019.","title":"How does disagreement help generalization against label corruption?","authors":"Yu","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhang and Sabuncu, 2018] Zhilu Zhang and Mert Sabuncu. Gen- eralized cross entropy loss for training deep neural networks with noisy labels. In NIPS, pages 8778-8788, 2018.","title":"Generalized cross entropy loss for training deep neural networks with noisy labels","authors":"Sabuncu Zhang, Zhilu Zhang, Mert Sabuncu","year":"2018","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhang et al., 2018] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.","title":"mixup: Beyond empirical risk minimization","authors":"Hongyi Zhang, Moustapha Cisse, David Yann N Dauphin, Lopez-Paz","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhang et al., 2020] Yikai Zhang, Songzhu Zheng, Pengxiang Wu, Mayank Goswami, and Chao Chen. Learning with feature- dependent label noise: A progressive approach. In International Conference on Learning Representations, 2020.","title":"Learning with featuredependent label noise: A progressive approach","authors":"Zhang","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zhang et al., 2021] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.","title":"Understanding deep learning (still) requires rethinking generalization","authors":"Zhang","year":"2021","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Zheng et al., 2020] Songzhu Zheng, Pengxiang Wu, Aman Goswami, Mayank Goswami, Dimitris Metaxas, and Chao Chen. Error-bounded correction of noisy labels. In ICML, pages 11447-11457. PMLR, 2020.","title":"Error-bounded correction of noisy labels","authors":"Zheng","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}