{"type":"ScholarlyArticle","title":"Label Distribution for Learning with Noisy Labels","authors":[{"name":"Yun-Peng Liu","slug":"yun-peng-liu","given":"Yun-Peng","family":"Liu","affiliation":"MOE Key Laboratory of Computer Network and Information Integration , China School of Computer Science and Engineering , Southeast University , Nanjing 210096 , China"},{"name":"Ning Xu","slug":"ning-xu","given":"Ning","family":"Xu","affiliation":"MOE Key Laboratory of Computer Network and Information Integration , China School of Computer Science and Engineering , Southeast University , Nanjing 210096 , China"},{"name":"Yu Zhang","slug":"yu-zhang","given":"Yu","family":"Zhang","affiliation":"MOE Key Laboratory of Computer Network and Information Integration , China School of Computer Science and Engineering , Southeast University , Nanjing 210096 , China"},{"name":"Xin Geng","slug":"xin-geng","given":"Xin","family":"Geng","affiliation":"MOE Key Laboratory of Computer Network and Information Integration , China School of Computer Science and Engineering , Southeast University , Nanjing 210096 , China"}],"abstract":"The performances of deep neural networks (DNNs) crucially rely on the quality of labeling. In some situations, labels are easily corrupted, and therefore some labels become noisy labels. Thus, designing algorithms that deal with noisy labels is of great importance for learning robust DNNs. However, it is difficult to distinguish between clean labels and noisy labels, which becomes the bottleneck of many methods. To address the problem, this paper proposes a novel method named Label Distribution based Confidence Estimation (LDCE). LDCE estimates the confidence of the observed labels based on label distribution. Then, the boundary between clean labels and noisy labels becomes clear according to confidence scores. To verify the effectiveness of the method, LDCE is combined with the existing learning algorithm to train robust DNNs. Experiments on both synthetic and real-world datasets substantiate the superiority of the proposed algorithm against state-of-the-art methods.","doi":"10.24963/ijcai.2020/356","url":"https://ijcai.org/proceedings/2020/356","pdf":"https://ijcai.org/proceedings/2020/0356.pdf","markdown":"https://ijcai.org/proceedings/2020/356.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/356","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":356,"pages":{"first":2568,"last":2574},"keywords":["Machine Learning: Deep Learning","Machine Learning: Classification"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24136/file_24136.m3u8","length_sec":271,"provider":"hls"}],"references":[{"position":1,"raw":"[Chen et al., 2019] Pengfei Chen, Ben Ben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. In Inter- national Conference on Machine Learning, pages 1062- 1070, 2019.","title":"Understanding and utilizing deep neural networks trained with noisy labels","authors":"Pengfei Chen, Ben Ben Liao, Guangyong Chen, Shengyu Zhang","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Divvala et al., 2014] Santosh K Divvala, Ali Farhadi, and Carlos Guestrin. Learning everything about anything: Webly-supervised visual concept learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3270-3277, 2014.","title":"Learning everything about anything: Webly-supervised visual concept learning","authors":"Ali Santosh K Divvala, Carlos Farhadi, Guestrin","year":"2014","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Geng, 2016] Xin Geng. Label distribution learning. IEEE Transactions on Knowledge and Data Engineering, 28(7):1734-1748, 2016.","title":"Label distribution learning","authors":"Xin Geng, Geng","year":"2016","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Guo et al., 2018] Sheng Guo, Weilin Huang, Haozhi Zhang, Chenfan Zhuang, Dengke Dong, Matthew R Scott, and Dinglong Huang. Curriculumnet: Weakly supervised learning from large-scale web images. In Proceedings of the European Conference on Computer Vision, pages 135- 150, 2018.","title":"Curriculumnet: Weakly supervised learning from large-scale web images","authors":"Sheng Guo, Weilin Huang, Haozhi Zhang, Chenfan Zhuang, Dengke Dong, Matthew R Scott, Dinglong Huang","year":"2018","venue":"Proceedings of the European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Han et al., 2018] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in neural information processing systems, pages 8527-8537, 2018.","title":"Co-teaching: Robust training of deep neural networks with extremely noisy labels","authors":"Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, Masashi Sugiyama","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hendrycks et al., 2018] Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train deep networks on labels corrupted by severe noise. In Advances in neural information processing systems, pages 10456-10465, 2018.","title":"Using trusted data to train deep networks on labels corrupted by severe noise","authors":"Dan Hendrycks, Mantas Mazeika, Duncan Wilson, Kevin Gimpel","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jiang et al., 2018] Lu Jiang, Zhengyuan Zhou, Thomas Le- ung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven curriculum for very deep neural networks on cor- rupted labels. In International Conference on Machine Learning, pages 2309-2318, 2018.","title":"Mentornet: Learning datadriven curriculum for very deep neural networks on corrupted labels","authors":"Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, Li Fei-Fei","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Krizhevsky et al., 2009] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny im- ages. Technical report, Citeseer, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lee et al., 2018] Kuang-Huei Lee, Xiaodong He, Lei Zhang, and Linjun Yang. Cleannet: Transfer learning for scalable image classifier training with label noise. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5447-5456, 2018.","title":"Cleannet: Transfer learning for scalable image classifier training with label noise","authors":"Kuang-Huei Lee, Xiaodong He, Lei Zhang, Linjun Yang","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2017] Yuncheng Li, Jianchao Yang, Yale Song, Liangliang Cao, Jiebo Luo, and Li-Jia Li. Learning from noisy labels with distillation. In Proceedings of the IEEE International Conference on Computer Vision, pages 1910-1918, 2017.","title":"Learning from noisy labels with distillation","authors":"Yuncheng Li, Jianchao Yang, Yale Song, Liangliang Cao, Jiebo Luo, Li-Jia Li","year":"2017","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Nettleton et al., 2010] David F Nettleton, Albert Orriols- Puig, and Albert Fornells. A study of the effect of differ- ent types of noise on the precision of supervised learning techniques. Artificial intelligence review, 33(4):275-306, 2010.","title":"A study of the effect of different types of noise on the precision of supervised learning techniques","authors":"Albert David F Nettleton, Albert Orriols-Puig, Fornells","year":"2010","venue":"Artificial intelligence review","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Patrini et al., 2017] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, pages 1944-1952, 2017.","title":"Making deep neural networks robust to label noise: A loss correction approach","authors":"Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, Lizhen Qu","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ren et al., 2018] Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International Conference on Ma- chine Learning, pages 4331-4340, 2018.","title":"Learning to reweight examples for robust deep learning","authors":"Ren","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ruder, 2017] Sebastian Ruder. An overview of multi- task learning in deep neural networks. arXiv preprint arXiv:1706.05098, 2017.","title":"An overview of multitask learning in deep neural networks","authors":"Sebastian Ruder","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Shu et al., 2019] Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta- weight-net: Learning an explicit mapping for sample weighting. In Advances in Neural Information Processing Systems, pages 1917-1928, 2019.","title":"Metaweight-net: Learning an explicit mapping for sample weighting","authors":"Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, Deyu Meng","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Tanaka et al., 2018] Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5552-5560, 2018.","title":"Joint optimization framework for learning with noisy labels","authors":"Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, Kiyoharu Aizawa","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing sys- tems, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang and Yao, 2019] Yaqing Wang and Quanming Yao. Few-shot learning: A survey. arXiv preprint arXiv:1904.05046, 2019.","title":"Yaqing Wang and Quanming Yao. Few-shot learning: A survey","authors":"Yao Wang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Xiao et al., 2015] Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 2691-2699, 2015.","title":"Learning from massive noisy labeled data for image classification","authors":"Tong Xiao, Tian Xia, Yi Yang, Chang Huang, Xiaogang Wang","year":"2015","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Xu et al., 2018] Ning Xu, An Tao, and Xin Geng. Label en- hancement for label distribution learning. In Proceedings of the International Joint Conference on Artificial Intelli- gence, pages 2926-2932, Stockholm, Sweden, 2018.","title":"Label enhancement for label distribution learning","authors":"Ning Xu, An Tao, Xin Geng","year":"2018","venue":"Proceedings of the International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2018/406"},{"position":22,"raw":"[Yao et al., 2019] Jiangchao Yao, Hao Wu, Ya Zhang, Ivor W Tsang, and Jun Sun. Safeguarded dynamic label regression for noisy supervision. In Proceedings of the AAAI Conference on Artificial Intelligence, 2019.","title":"Safeguarded dynamic label regression for noisy supervision","authors":"Jiangchao Yao, Hao Wu, Ya Zhang, Ivor W Tsang, Jun Sun","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yi and Wu, 2019] Kun Yi and Jianxin Wu. Probabilistic end-to-end noise correction for learning with noisy labels. arXiv preprint arXiv:1903.07788, 2019.","title":"Probabilistic end-to-end noise correction for learning with noisy labels","authors":"Yi, Wu ; ] Kun Yi, Jianxin Wu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zagoruyko and Komodakis, 2016] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016.","title":"Wide residual networks","authors":"Sergey Zagoruyko, Nikos Komodakis","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}