{"type":"ScholarlyArticle","title":"MILD: Modeling the Instance Learning Dynamics for Learning with Noisy Labels","authors":[{"name":"Chuanyang Hu","slug":"chuanyang-hu","given":"Chuanyang","family":"Hu","affiliation":"ShanghaiTech University"},{"name":"Shipeng Yan","slug":"shipeng-yan","given":"Shipeng","family":"Yan","affiliation":"ShanghaiTech University"},{"name":"Zhitong Gao","slug":"zhitong-gao","given":"Zhitong","family":"Gao","affiliation":"ShanghaiTech University"},{"name":"Xuming He","slug":"xuming-he","given":"Xuming","family":"He","affiliation":"ShanghaiTech University; Shanghai Engineering Research Center of Intelligent Vision and Imaging"}],"abstract":"Despite deep learning has achieved great success, it often relies on a large amount of training data with accurate labels, which are expensive and time-consuming to collect. A prominent direction to reduce the cost is to learn with noisy labels, which are ubiquitous in the real-world applications. A critical challenge for such a learning task is to reduce the effect of network memorization on the falsely-labeled data.  In this work, we propose an iterative selection approach based on the Weibull mixture model, which identifies clean data by considering the overall learning dynamics of each data instance. In contrast to the previous small-loss heuristics, we leverage the observation that deep network is easy to memorize and hard to forget clean data. In particular, we measure the difficulty of memorization and forgetting for each instance via the transition times between being misclassified and being memorized in training, and integrate them into a novel metric for selection. Based on the proposed metric, we retain a subset of identified clean data and repeat the selection procedure to iteratively refine the clean subset, which is finally used for model training. To validate our method, we perform extensive experiments on synthetic noisy datasets and real-world web data, and our strategy outperforms existing noisy-label learning methods.","doi":"10.24963/ijcai.2023/92","url":"https://ijcai.org/proceedings/2023/92","pdf":"https://ijcai.org/proceedings/2023/0092.pdf","markdown":"https://ijcai.org/proceedings/2023/92.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/92","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":92,"pages":{"first":828,"last":836},"keywords":["Computer Vision: CV: Machine learning for vision","Computer Vision: CV: Recognition (object detection, categorization)"],"videos":[],"references":[{"position":1,"raw":"[Arazo et al., 2019] Eric Arazo, Diego Ortego, Paul Albert, Noel O'Connor, and Kevin McGuinness. Unsupervised label noise modeling and loss correction. In International conference on machine learning(ICML), 2019.","title":"Unsupervised label noise modeling and loss correction","authors":"Eric Arazo, Diego Ortego, Paul Albert, O' Noel, Kevin Connor, Mcguinness","year":"2019","venue":"International conference on machine learning(ICML)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Arpit et al., 2017] Devansh Arpit, Stanisław Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memo- rization in deep networks. In International conference on machine learning(ICML), 2017.","title":"A closer look at memorization in deep networks","authors":"Devansh Arpit, Stanisław Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, S Maxinder, Tegan Kanwal, Asja Maharaj, Aaron Fischer, Yoshua Courville, Bengio","year":"2017","venue":"International conference on machine learning(ICML)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Berthelot et al., 2019] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi- supervised learning. Advances in Neural Information Pro- cessing Systems, 2019.","title":"Mixmatch: A holistic approach to semisupervised learning","authors":"David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, Colin A Raffel","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Blum et al., 2003] Avrim Blum, Adam Kalai, and Hal Wasserman. Noise-tolerant learning, the parity problem, and the statistical query model. Journal of the ACM (JACM), 2003.","title":"Noise-tolerant learning, the parity problem, and the statistical query model","authors":"Avrim Blum, Adam Kalai, Hal Wasserman","year":"2003","venue":"Journal of the ACM","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2019] Pengfei Chen, Ben Ben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. In Inter- national Conference on Machine Learning(ICML), 2019.","title":"Understanding and utilizing deep neural networks trained with noisy labels","authors":"Pengfei Chen, Ben Ben Liao, Guangyong Chen, Shengyu Zhang","year":"2019","venue":"International Conference on Machine Learning(ICML)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Feng et al., 2020] Di Feng, Christian Haase-Schütz, Lars Rosenbaum, Heinz Hertlein, Claudius Glaeser, Fabian Timm, Werner Wiesbeck, and Klaus Dietmayer. Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and chal- lenges. IEEE Transactions on Intelligent Transportation Systems, 2020.","title":"Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges","authors":"Di Feng, Christian Haase-Schütz, Lars Rosenbaum, Heinz Hertlein, Claudius Glaeser, Fabian Timm, Werner Wiesbeck, Klaus Dietmayer","year":"2020","venue":"IEEE Transactions on Intelligent Transportation Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ghosh et al., 2017] Aritra Ghosh, Himanshu Kumar, and PS Sastry. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI conference on artificial intelligence(AAAI), 2017.","title":"Robust loss functions under label noise for deep neural networks","authors":"Aritra Ghosh, Himanshu Kumar, Sastry","year":"2017","venue":"Proceedings of the AAAI conference on artificial intelligence(AAAI)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Han et al., 2018a] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems(NeurIPS, 2018.","title":"Co-teaching: Robust training of deep neural networks with extremely noisy labels","authors":"Han","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Han et al., 2018b] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems, 31, 2018.","title":"Co-teaching: Robust training of deep neural networks with extremely noisy labels","authors":"Han","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jiang et al., 2018] Lu Jiang, Zhengyuan Zhou, Thomas Le- ung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven curriculum for very deep neural networks on cor- rupted labels. In International Conference on Machine Learning(ICML), 2018.","title":"Mentornet: Learning datadriven curriculum for very deep neural networks on corrupted labels","authors":"Jiang","year":"2018","venue":"International Conference on Machine Learning(ICML)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jiang et al., 2020] Lu Jiang, Di Huang, Mason Liu, and Weilong Yang. Beyond synthetic noise: Deep learning on controlled noisy labels. In International Conference on Machine Learning(ICML), 2020.","title":"Beyond synthetic noise: Deep learning on controlled noisy labels","authors":"Lu Jiang, Di Huang, Mason Liu, Weilong Yang","year":"2020","venue":"International Conference on Machine Learning(ICML)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kim et al., 2021] Taehyeon Kim, Jongwoo Ko, JinHwan Choi, Se-Young Yun, et al. Fine samples for learning with noisy labels. Advances in Neural Information Processing Systems, 34, 2021.","title":"Fine samples for learning with noisy labels","authors":"Kim","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Krizhevsky et al., 2009] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny im- ages. Technical report, University of Toronto, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2017] Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. Webvision database: Visual learning and understanding from web data. arXiv preprint arXiv:1708.02862, 2017.","title":"Webvision database: Visual learning and understanding from web data","authors":"Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, Luc Van Gool","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2020] Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi- supervised learning. arXiv preprint arXiv:2002.07394, 2020.","title":"Dividemix: Learning with noisy labels as semisupervised learning","authors":"Li","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2021] Shuailin Li, Zhitong Gao, and Xuming He. Superpixel-guided iterative learning from noisy labels for medical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted In- tervention(MICCAI), 2021.","title":"Superpixel-guided iterative learning from noisy labels for medical image segmentation","authors":"Li","year":"2021","venue":"International Conference on Medical Image Computing and Computer-Assisted Intervention(MICCAI)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2020] Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernandez-Granda. Early-learning regularization prevents memorization of noisy labels. Advances in neural information processing systems, 33:20331-20342, 2020.","title":"Early-learning regularization prevents memorization of noisy labels","authors":"Sheng Liu, Jonathan Niles-Weed, Narges Razavian, Carlos Fernandez-Granda","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ma et al., 2020] Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bailey. Nor- malized loss functions for deep learning with noisy labels. In International Conference on Machine Learning(ICML), 2020.","title":"Normalized loss functions for deep learning with noisy labels","authors":"Ma","year":"2020","venue":"International Conference on Machine Learning(ICML)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mirzasoleiman et al., 2020] Baharan Mirzasoleiman, Kaidi Cao, and Jure Leskovec. Coresets for robust training of neural networks against noisy labels. arXiv preprint arXiv:2011.07451, 2020.","title":"Coresets for robust training of neural networks against noisy labels","authors":"Mirzasoleiman","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Nguyen et al., 2019] Duc Tam Nguyen, Chaithanya Kumar Mummadi, Thi Phuong Nhung Ngo, Thi Hoai Phuong Nguyen, Laura Beggel, and Thomas Brox. Self: Learning to filter noisy labels with self-ensembling. arXiv preprint arXiv:1910.01842, 2019.","title":"Self: Learning to filter noisy labels with self-ensembling","authors":"Duc Tam Nguyen, Chaithanya Kumar Mummadi, Thi Phuong, Nhung Ngo, Thi Hoai, Phuong Nguyen, Laura Beggel, Thomas Brox","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ortego et al., 2021] Diego Ortego, Eric Arazo, Paul Al- bert, Noel E O'Connor, and Kevin McGuinness. Multi- objective interpolation training for robustness to label noise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6606- 6615, 2021.","title":"Multiobjective interpolation training for robustness to label noise","authors":"Ortego","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Patrini et al., 2017] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE confer- ence on computer vision and pattern recognition, pages 1944-1952, 2017.","title":"Making deep neural networks robust to label noise: A loss correction approach","authors":"Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, Lizhen Qu","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Reed et al., 2014] Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy la- bels with bootstrapping. arXiv preprint arXiv:1412.6596, 2014.","title":"Training deep neural networks on noisy labels with bootstrapping","authors":"Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, Andrew Rabinovich","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Shen and Sanghavi, 2019] Yanyao Shen and Sujay Sang- havi. Learning with bad training data via iterative trimmed loss minimization. In International Conference on Ma- chine Learning(ICML), 2019.","title":"Learning with bad training data via iterative trimmed loss minimization","authors":"Sanghavi ; Sujay Shen, Sanghavi","year":"2019","venue":"International Conference on Machine Learning(ICML)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Song et al., 2019] Hwanjun Song, Minseok Kim, and Jae- Gil Lee. Selfie: Refurbishing unclean samples for robust deep learning. In International Conference on Machine Learning(ICML), 2019.","title":"Selfie: Refurbishing unclean samples for robust deep learning","authors":"Song","year":"2019","venue":"International Conference on Machine Learning(ICML)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Song et al., 2022] Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy la- bels with deep neural networks: A survey. IEEE Transac- tions on Neural Networks and Learning Systems(TNNLS), 2022.","title":"Learning from noisy labels with deep neural networks: A survey","authors":"Song","year":"2022","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Szegedy et al., 2017] Christian Szegedy, Sergey Ioffe, Vin- cent Vanhoucke, and Alexander A Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In Thirty-first AAAI conference on artificial in- telligence, 2017.","title":"Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning","authors":"Szegedy","year":"2017","venue":"Thirty-first AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Toneva et al., 2018] Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Ben- gio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learn- ing. In International Conference on Learning Represen- tations(ICLR), 2018.","title":"An empirical study of example forgetting during deep neural network learning","authors":"Mariya Toneva, Alessandro Sordoni, Remi Tachet Des Combes, Adam Trischler, Yoshua Bengio, Geoffrey J Gordon","year":"2018","venue":"International Conference on Learning Representations(ICLR)","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wei et al., 2020] Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 13726-13735, 2020.","title":"Combating noisy labels by agreement: A joint training method with co-regularization","authors":"Hongxin Wei, Lei Feng, Xiangyu Chen, Bo An","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wu et al., 2020] Pengxiang Wu, Songzhu Zheng, Mayank Goswami, Dimitris Metaxas, and Chao Chen. A topologi- cal filter for learning with label noise. Advances in neural information processing systems, 33:21382-21393, 2020.","title":"A topological filter for learning with label noise","authors":"Pengxiang Wu, Songzhu Zheng, Mayank Goswami, Dimitris Metaxas, Chao Chen","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yan et al., 2014] Yan Yan, Rómer Rosales, Glenn Fung, Ramanathan Subramanian, and Jennifer Dy. Learning from multiple annotators with varying expertise. Machine learning, 2014.","title":"Learning from multiple annotators with varying expertise","authors":"Yan Yan, Rómer Rosales, Glenn Fung, Ramanathan Subramanian, Jennifer Dy","year":"2014","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yu et al., 2019] Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does dis- agreement help generalization against label corruption? In International Conference on Machine Learning, pages 7164-7173. PMLR, 2019.","title":"How does disagreement help generalization against label corruption?","authors":"Yu","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhang and Sabuncu, 2018] Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural informa- tion processing systems(NeurIPS), 2018.","title":"Generalized cross entropy loss for training deep neural networks with noisy labels","authors":"Sabuncu Zhang, Zhilu Zhang, Mert Sabuncu","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhang et al., 2017a] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412, 2017.","title":"mixup: Beyond empirical risk minimization","authors":"Zhang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhang et al., 2017b] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412, 2017.","title":"mixup: Beyond empirical risk minimization","authors":"Zhang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhou et al., 2020] Tianyi Zhou, Shengjie Wang, and Jeff Bilmes. Robust curriculum learning: from clean label de- tection to noisy label self-correction. In International Con- ference on Learning Representations, 2020.","title":"Robust curriculum learning: from clean label detection to noisy label self-correction","authors":"Zhou","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhu et al., 2021] Jianing Zhu, Jingfeng Zhang, Bo Han, Tongliang Liu, Gang Niu, Hongxia Yang, Mohan Kankan- halli, and Masashi Sugiyama. Understanding the inter- action of adversarial training with noisy labels. arXiv preprint arXiv:2102.03482, 2021.","title":"Understanding the interaction of adversarial training with noisy labels","authors":"Jianing Zhu, Jingfeng Zhang, Bo Han, Tongliang Liu, Gang Niu, Hongxia Yang, Mohan Kankanhalli, Masashi Sugiyama","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}