{"type":"ScholarlyArticle","title":"Can Cross Entropy Loss Be Robust to Label Noise?","authors":[{"name":"Lei Feng","slug":"lei-feng","given":"Lei","family":"Feng","affiliation":"School of Computer Science and Engineering , Nanyang Technological University , Singapore"},{"name":"Senlin Shu","slug":"senlin-shu","given":"Senlin","family":"Shu","affiliation":"College of Computer and Information Science , Southwest University , Chongqing , China"},{"name":"Zhuoyi Lin","slug":"zhuoyi-lin","given":"Zhuoyi","family":"Lin","affiliation":"School of Computer Science and Engineering , Nanyang Technological University , Singapore"},{"name":"Fengmao Lv","slug":"fengmao-lv","given":"Fengmao","family":"Lv","affiliation":"Center of Statistical Research , Southwestern University of Finance and Economics , China"},{"name":"Li Li","slug":"li-li","given":"Li","family":"Li","affiliation":"College of Computer and Information Science , Southwest University , Chongqing , China"},{"name":"Bo An","slug":"bo-an","given":"Bo","family":"An","affiliation":"School of Computer Science and Engineering , Nanyang Technological University , Singapore"}],"abstract":"Trained with the standard cross entropy loss, deep neural networks can achieve great performance on correctly labeled data. However, if the training data is corrupted with label noise, deep models tend to overfit the noisy labels, thereby achieving poor generation performance. To remedy this issue, several loss functions have been proposed and demonstrated to be robust to label noise. Although most of the robust loss functions stem from Categorical Cross Entropy (CCE) loss, they fail to embody the intrinsic relationships between CCE and other loss functions. In this paper, we propose a general framework dubbed Taylor cross entropy loss to train deep models in the presence of label noise. Specifically, our framework enables to weight the extent of fitting the training labels by controlling the order of Taylor Series for CCE, hence it can be robust to label noise. In addition, our framework clearly reveals the intrinsic relationships between CCE and other loss functions, such as Mean Absolute Error (MAE) and Mean Squared Error (MSE). Moreover, we present a detailed theoretical analysis to certify the robustness of this framework. Extensive experimental results on benchmark datasets demonstrate that our proposed approach significantly outperforms the state-of-the-art counterparts.","doi":"10.24963/ijcai.2020/305","url":"https://ijcai.org/proceedings/2020/305","pdf":"https://ijcai.org/proceedings/2020/0305.pdf","markdown":"https://ijcai.org/proceedings/2020/305.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/305","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":305,"pages":{"first":2206,"last":2212},"keywords":["Machine Learning: Classification","Data Mining: Classification, Semi-Supervised Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24324/file_24324.m3u8","length_sec":223,"provider":"hls"}],"references":[{"position":1,"raw":"[Berthon et al., 2020] Antonin Berthon, Bo Han, Gang Ni- u, Tongliang Liu, and Masashi Sugiyama. Confidence s- cores make instance-dependent label-noise learning possi- ble. arXiv preprint arXiv:2001.03772, 2020.","title":"Confidence scores make instance-dependent label-noise learning possible","authors":"Antonin Berthon, Bo Han, Gang Niu, Tongliang Liu, Masashi Sugiyama","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Clanuwat et al., 2018] Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv preprint arXiv:1812.01718, 2018.","title":"Deep learning for classical japanese literature","authors":"Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, David Ha","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ghosh et al., 2015] Aritra Ghosh, Naresh Manwani, and P- S Sastry. Making risk minimization tolerant to label noise. Neurocomputing, 160:93-107, 2015.","title":"Making risk minimization tolerant to label noise","authors":"Aritra Ghosh, Naresh Manwani, P-S Sastry","year":"2015","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Ghosh et al., 2017] Aritra Ghosh, Himanshu Kumar, and P- S Sastry. Robust loss functions under label noise for deep neural networks. In AAAI, 2017.","title":"Robust loss functions under label noise for deep neural networks","authors":"Aritra Ghosh, Himanshu Kumar, P-S Sastry","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Goldberger and Ben-Reuven, 2017] Jacob Goldberger and Ehud Ben-Reuven. Training deep neural-networks using a noise adaptation layer. In ICLR, 2017.","title":"Training deep neural-networks using a noise adaptation layer","authors":"Ben-Reuven ; Goldberger, Jacob Goldberger, Ehud Ben-Reuven","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Han et al., 2018a] Bo Han, Gang Niu, Jiangchao Yao, X- ingrui Yu, Miao Xu, Ivor Tsang, and Masashi Sugiyama. Pumpout: A meta approach to robust deep learning with noisy labels. arXiv preprint arXiv:1809.11008, 2018.","title":"Pumpout: A meta approach to robust deep learning with noisy labels","authors":"Han","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Han et al., 2018b] Bo Han, Jiangchao Yao, Gang Niu, Mingyuan Zhou, Ivor Tsang, Ya Zhang, and Masashi Sugiyama. Masking: A new perspective of noisy super- vision. In NuerIPS, pages 5836-5846, 2018.","title":"Masking: A new perspective of noisy supervision","authors":"Han","year":"2018","venue":"NuerIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Han et al., 2018c] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NuerIPS, pages 8527-8537, 2018.","title":"Co-teaching: Robust training of deep neural networks with extremely noisy labels","authors":"Han","year":"2018","venue":"NuerIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hendrycks et al., 2018] Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train deep networks on labels corrupted by severe noise. In NuerIPS, pages 10456-10465, 2018.","title":"Using trusted data to train deep networks on labels corrupted by severe noise","authors":"Dan Hendrycks, Mantas Mazeika, Duncan Wilson, Kevin Gimpel","year":"2018","venue":"NuerIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jiang et al., 2017] Lu Jiang, Zhengyuan Zhou, Thomas Le- ung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven curriculum for very deep neural networks on cor- rupted labels. arXiv preprint arXiv:1712.05055, 2017.","title":"Mentornet: Learning datadriven curriculum for very deep neural networks on corrupted labels","authors":"Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, Li Fei-Fei","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Khetan et al., 2017] Ashish Khetan, Zachary C Lipton, and Anima Anandkumar. Learning from noisy singly-labeled data. arXiv preprint arXiv:1712.04577, 2017.","title":"Learning from noisy singly-labeled data","authors":"Ashish Khetan, Zachary C Lipton, Anima Anandkumar","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Krizhevsky et al., 2009] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny im- ages. Technical Report, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[LeCun et al., 1998] Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning ap- plied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Yann Lecun, Léon Bottou, Yoshua Bengio, Patrick Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Menon et al., 2015] Aditya Menon, Brendan Van Rooyen, Cheng Soon Ong, and Bob Williamson. Learning from corrupted binary labels via class-probability estimation. In ICML, pages 125-134, 2015.","title":"Learning from corrupted binary labels via class-probability estimation","authors":"Aditya Menon, Brendan Van Rooyen, Cheng, Soon Ong, Bob Williamson","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Menon et al., 2020] Aditya Krishna Menon, Ankit Singh Rawat, Sashank J. Reddi, and Sanjiv Kumar. Can gradient clipping mitigate label noise? In ICLR, 2020.","title":"Can gradient clipping mitigate label noise?","authors":"Aditya Krishna Menon, Ankit Singh Rawat, Sashank J Reddi, Sanjiv Kumar","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Natarajan et al., 2013] Nagarajan Natarajan, Inderjit S D- hillon, Pradeep K Ravikumar, and Ambuj Tewari. Learn- ing with noisy labels. In NuerIPS, pages 1196-1204, 2013.","title":"Learning with noisy labels","authors":"Nagarajan Natarajan, Pradeep K Inderjit S Dhillon, Ambuj Ravikumar, Tewari","year":"2013","venue":"NuerIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Patrini et al., 2017] Giorgio Patrini, Alessandro Rozza, A- ditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In CVPR, pages 1944-1952, 2017.","title":"Making deep neural networks robust to label noise: A loss correction approach","authors":"Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, Lizhen Qu","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sukhbaatar and Fergus, 2014] Sainbayar Sukhbaatar and Rob Fergus. Learning from noisy labels with deep neural networks. arXiv preprint arXiv:1406.2080, 2(3):4, 2014.","title":"Learning from noisy labels with deep neural networks","authors":"Fergus ; Sukhbaatar, Sainbayar Sukhbaatar, Rob Fergus","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tanaka et al., 2018] Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. In CVPR, pages 5552-5560, 2018.","title":"Joint optimization framework for learning with noisy labels","authors":"Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, Kiyoharu Aizawa","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2019] Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In ICCV, pages 322-330, 2019.","title":"Symmetric cross entropy for robust learning with noisy labels","authors":"Wang","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wei et al., 2020] Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A join- t training method with co-regularization. arXiv preprint arXiv:2003.02752v3, 2020.","title":"Combating noisy labels by agreement: A joint training method with co-regularization","authors":"Hongxin Wei, Lei Feng, Xiangyu Chen, Bo An","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xia et al., 2019] Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learn- ing? In NuerIPS, pages 6835-6846, 2019.","title":"Are anchor points really indispensable in label-noise learning? In NuerIPS","authors":"Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, Masashi Sugiyama","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xiao et al., 2017] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: A novel image dataset for bench- marking machine learning algorithms. arXiv preprint arX- iv:1708.07747, 2017.","title":"Fashion-mnist: A novel image dataset for benchmarking machine learning algorithms","authors":"Han Xiao, Kashif Rasul, Roland Vollgraf","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yang et al., 2019] Hansi Yang, Quanming Yao, Bo Han, and Gang Niu. Searching to exploit memorization effec- t in learning from corrupted labels. arXiv preprint arX- iv:1911.02377, 2019.","title":"Searching to exploit memorization effect in learning from corrupted labels","authors":"Hansi Yang, Quanming Yao, Bo Han, Gang Niu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yi and Wu, 2019] Kun Yi and Jianxin Wu. Probabilistic end-to-end noise correction for learning with noisy labels. In CVPR, pages 7017-7025, 2019.","title":"Probabilistic end-to-end noise correction for learning with noisy labels","authors":"Yi, Wu ; ] Kun Yi, Jianxin Wu","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yu et al., 2019] Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor W Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In ICML, pages 7164-7173, 2019.","title":"How does disagreement help generalization against label corruption?","authors":"Yu","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang and Sabuncu, 2018] Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In NuerIPS, pages 8778-8788, 2018.","title":"Generalized cross entropy loss for training deep neural networks with noisy labels","authors":"Sabuncu Zhang, Zhilu Zhang, Mert Sabuncu","year":"2018","venue":"NuerIPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}