--- title: "Can Cross Entropy Loss Be Robust to Label Noise?" authors: - "Lei Feng" - "Senlin Shu" - "Zhuoyi Lin" - "Fengmao Lv" - "Li Li" - "Bo An" doi: "10.24963/ijcai.2020/305" url: "https://ijcai.org/proceedings/2020/305" pdf: "https://ijcai.org/proceedings/2020/0305.pdf" venue: "Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence" year: 2020 pages: "2206-2212" track: "Main track" keywords: - "Machine Learning: Classification" - "Data Mining: Classification, Semi-Supervised Learning" --- # Can Cross Entropy Loss Be Robust to Label Noise? Lei Feng, Senlin Shu, Zhuoyi Lin, Fengmao Lv, Li Li, Bo An *Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence*, pages 2206-2212. DOI: [10.24963/ijcai.2020/305](https://doi.org/10.24963/ijcai.2020/305) ## Abstract Trained with the standard cross entropy loss, deep neural networks can achieve great performance on correctly labeled data. However, if the training data is corrupted with label noise, deep models tend to overfit the noisy labels, thereby achieving poor generation performance. To remedy this issue, several loss functions have been proposed and demonstrated to be robust to label noise. Although most of the robust loss functions stem from Categorical Cross Entropy (CCE) loss, they fail to embody the intrinsic relationships between CCE and other loss functions. In this paper, we propose a general framework dubbed Taylor cross entropy loss to train deep models in the presence of label noise. Specifically, our framework enables to weight the extent of fitting the training labels by controlling the order of Taylor Series for CCE, hence it can be robust to label noise. In addition, our framework clearly reveals the intrinsic relationships between CCE and other loss functions, such as Mean Absolute Error (MAE) and Mean Squared Error (MSE). Moreover, we present a detailed theoretical analysis to certify the robustness of this framework. Extensive experimental results on benchmark datasets demonstrate that our proposed approach significantly outperforms the state-of-the-art counterparts. ## Full text ## 1 Introduction Deep Neural Networks (DNNs) have achieved great advances over the past years. With proper training, DNNs can easily achieve great classification performance. However, the success of DNNs relies on a large number of high-quality samples during the training process. Unfortunately, incorrect labels in large-scale datasets are often inevitable. In most scenarios, it can be more beneficial to have datasets with more but noisier labels than less but more accurate labels [Khetan et al., 2017]. Therefore, training a robust classifier in the presence of label noise is an increasingly valued task. In general, (softmax) Categorical Cross Entropy (CCE) loss is the standard loss function used to train deep models. However, past studies [Ghosh et al., 2017; Patrini et al., 2017] show that using the standard CCE to train deep models leads to serious over-fitting (on noisy labels) and results in poor generalization ability. To mitigate this issue, increasing interests have been drawn in exploiting robust loss functions for training deep models against label noise. For example, the works of [Ghosh et al., 2015; Ghosh et al., 2017] reveal that symmetric loss functions, in which the sum of the risks over all categories is equivalent to a constant for each arbitrary example, can be robust to label noise. Representative symmetric loss functions include Ramp Loss [Ghosh et al., 2015] and (softmax) Mean Absolute Error (MAE) [Ghosh et al., 2017]. By both theoretical and empirical analysis, symmetric loss functions are demonstrated to be robust to label noise. Recently, several other loss functions that do not satisfy the symmetry condition strictly, including Generalized Cross Entropy (GCE) loss [Zhang and Sabuncu, 2018], Partial Huberised Cross Entropy (PHuberCE) loss [Menon et al., 2020] and Symmetric Cross Entropy (SCE) loss [Wang et al., 2019], have also been proposed to be robust against label noise when training deep neural networks. Although these loss functions achieve robustness to label noise in different ways, one thing in common is that these loss functions are derived from the standard CCE. In spite of their effectiveness, they all fail to embody the intrinsic relationships between CCE and other loss functions. ∗Corresponding author. Motivated by the above observations, we wonder whether CCE can be further exploited to design a general framework that embody the intrinsic relationships between CCE and other loss functions, for robust learning with label noise. To answer this question, this paper proposes a general robust learning framework to train deep models in the presence of label noise. Specifically, we apply Taylor Series to derive an alternative representation of CCE. Moreover, we can flexibly adjust the order of Taylor Series to approximate CCE, and we call the approximated CCE Taylor Cross Entropy (TCE). In this framework, the order of Taylor Series reflects the extent of how TCE approximates to CCE. Furthermore, by varying the order of Taylor Series, we are able to reveal the intrinsic relationships between CCE and other loss functions. For example, MAE can be considered as the first-order Taylor Series approximation of CCE. The second-order Taylor Series approximation of CCE is an average combination of MAE and a lower bound of Mean Squared Error (MSE). The sufficiently large-order Taylor Series approximation of CCE recovers CCE. In addition, we show that our proposed TCE is upperbounded with finite order (which means the risk would never be infinite), and TCE is always an upper bound of MAE. In other words, minimizing TCE naturally enables to minimize MAE to some degree. To sum up, our main contributions are three-fold: - We propose a general robust learning framework dubbed Taylor Cross Entropy (TCE) loss. Taylor Series is applied to obtain a representation of CCE for training deep models in the presence of label noise. - We present a detailed theoretical analysis to certify the robustness of TCE against label noise. - We conduct extensive experiments for learning from symmetric and asymmetric label noise, and compare with a number of loss functions. Extensive experimental results on benchmark datasets demonstrate that our proposed approach significantly outperforms the stateof-the-art counterparts. ## 2 RelatedWork In this section, we briefly review existing works on learning in the presence of label noise. Noise rate estimation. Some of the early works [Natarajan et al., 2013; Sukhbaatar and Fergus, 2014; Menon et al., 2015; Patrini et al., 2017] aim to estimate the label transition matrix (sometimes called confusion matrix), and use it to train the target model. For this type of approach, the classification performance hinges on the quality of noise rate estimation [Goldberger and Ben-Reuven, 2017; Hendrycks et al., 2018; Han et al., 2018b; Xia et al., 2019]. However, noise rate estimation is challenging, especially on datasets with a huge number of classes. Robust loss functions. Designing loss functions that are robust to label noise has been received increasing attention from researchers. The first work is from [Ghosh et al., 2015], which demonstrates that binary loss functions that satisfy the symmetric condition ℓ(z) + ℓ(-z) = c (e.g., ramp loss and sigmoid loss) where c is a constant, are robust to label noise for binary classification. Then, for multi-class classification, loss functions that satisfy the symmetric condition ∑kj=1 L(f(x), j) = C (e.g., MAE) where C is a constant, are demonstrated to be robust to label noise for the multiclass classification [Ghosh et al., 2017]. However, a recent study [Zhang and Sabuncu, 2018] shows that MAE is not able to achieve good performance on complicated datasets, due to its optimization issue. To alleviate this problem, Generalized Cross Entropy (GCE) [Zhang and Sabuncu, 2018] adopts the negative Box-Cox transformation strategy, and uses a hyperparameter q to balance between MAE and CCE. Partial Huberised Cross Entropy (PHuber-CE) [Menon et al., 2020] corrects CCE on hard examples by gradient clipping. Symmetric Cross Entropy (SCE) [Wang et al., 2019] combines CCE and Reverse Cross Entropy (RCE, which is equivalent to MAE) by tuning the regularization parameters. Although the above loss functions stem from CCE and some of them may be able to recover MAE or CCE by tuning the parameters, they all fail to embody the intrinsic relationships between CCE and other loss functions. Other deep learning methods. There are some other approaches that adopt other solutions [Wei et al., 2020; Tanaka et al., 2018; Han et al., 2018a; Yu et al., 2019; Berthon et al., 2020; Yang et al., 2019] to deal with noisy labels. For example, MentorNet [Jiang et al., 2017] is trained to supervise the training of a StudentNet with a sample weighting scheme. Co-teaching [Han et al., 2018c] trains two networks simultaneously and enables the two networks to learn from each other. PENCIL [Yi and Wu, 2019] trains neural networks using label probability distributions and updates these distributions in each epoch. ## 3 Taylor Cross Entropy Loss for Robust Learning with Label Noise In this section, we first briefly review CCE and MAE. Then, we introduce our proposed Taylor cross entropy loss. Finally, we theoretically analyze the robustness of Taylor cross entropy loss. ## 3.1 Preliminaries We consider the problem of k-class classification. Suppose the clean data set is represented as D = {(xi, yi) | 1 ≤ i ≤ m}, where xi ∈ X (X ∈ Rd ) is a d-dimensional feature vector and yi ∈ {1, . . . , k} is the label associated with xi. A classifier is a function that maps the feature space to the label space f : X → Rk . In this paper, we consider the common case where the function f is a DNN with the softmax output layer. In this way, the commonly used loss functions CCE and MAE can be represented as: where the fy(x) denotes the y-th element of f(x) and ey is a one-hot vector with eyj = 1 if j = y, otherwise 0. Besides, the gradients of CCE and MAE can be shown as: where θ is the set of parameters of f. As shown in the above equations, when training with CCE, examples with smaller prediction confidences are weighted more than examples with larger prediction confidences for gradient update. In other words, CCE pays more attention to hard examples. On the contrary, MAE treats all the examples equally. Hence CCE is more preferred than MAE when training with clean data. However, in the presence of label noise, the performance of CCE dramatically drops, since the given labels of hard examples may be incorrect. In the extreme case, the weight becomes infinite if fy(x) → 0, which leads to serious overfitting on noisy labels. In contrast, MAE does not suffer from this problem, since each example is treated equally. Therefore, there arises a question: can cross entropy loss be robust to label noise? To answer this question, we propose a general framework of CCE that endows the robustness to CCE against label noise. Figure 1: Lt-CE with different parameters t. ![Image](https://ijcai.org/proceedings/2020/fulltext/305/image_000000_b4ae89b560f28692d10f9916e6d83610bfa4c5aa64b29d0417fa88ebdafe0d48.png) ## 3.2 Taylor Cross Entropy Loss Given a function g(x), if g(x) is differentiable to order n at x = x0, then g(x) can be written as a Taylor Series: where gi (x0) denotes the i-th order derivative of g(x) at x0. Recall that LCCE(f(x), y) = -log fy(x), we can define g(fy(x)) = - log fy(x). Then we have If we set fy(x0) = 1, then ∀i ≥ 1 we have Thus we can express LCCE as Obviously, it is unrealistic to take into account all the terms, since n → ∞. Hence we propose to reserve finite terms in Eq. (3) and obtain an approximation of CCE, which is called Taylor Cross Entropy (TCE) loss: where t ∈ N+ is a hyper-parameter that denotes the order of the Taylor Series, in other words, the proximity to the CCE. Figure 1 illustrates Lt-CE with t varying in {1, . . . , 10}. According to the definition of Lt-CE and Figure 1, we can obtain some interesting deductions. Theorem 1. Our proposed Taylor cross entropy loss has the following properties: 1) When t = 1, Lt-CE = 1 2 LMAE; 2) When t = 2, Lt-CE is an average combination of LMAE and a lower bound of LMSE; 3) When t →∞, Lt-CE is equivalent to LCCE; 4) LMAE ≤ 2Lt-CE, ∀t ∈ N+. Proof. 1) When t = 1, Lt-CE is represented as It is clear that in this case, Lt-CE = 12 LMAE, which means, minimizing Lt-CE is equivalent to minimizing LMAE. - 2) When t = 2, Lt-CE is represented as For the first term, we have already shown (1 - fy(x)) = 12 LMAE. For the second term, we have Hence the second term is a lower bound of MSE. - 3) According to the definition of Lt-CE, it is clear that Lt-CE will be equivalent to LCCE if t →∞. - 4) Based on Property 1), we have Since the second term is always non-negative, LMAE ≤ 2Lt-CE(f(x), y), and the equality holds when and only when t = 1. However, the difference becomes larger as t increases, which also suggests that LCCE is not robust from another point of view, since LMAE is the standard robust loss with strong theoretic guarantees [Ghosh et al., 2017]. Properties 1) and 3) show that LMAE and LCE can be considered as special cases of Lt-CE. Properties 2) and 4) certify the robustness of Lt-CE to some degree, since LMAE and LMSE have been shown to be robust in the presence of label noise [Ghosh et al., 2017]. ## 3.3 Theoretical Analysis Here, we theoretically analyze the robustness of our proposed Taylor cross entropy loss. By the definition of Lt-CE (i.e., Eq. (4)), we can easily derive an upper bound and a lower bound of Lt-CE as follows: Lemma 1. For any x and any positive integer t < +∞, the sum of Lt-CE with respect to all the classes satisfies: where Ct = ∑ti=1 1i is a constant that depends on t. Proof. Based on Eq. (5), if we consider the sum of Lt-CE with respect to all the classes, the following equality holds: Hence which concludes the proof. We can find that Lt-CE is not always a symmetric loss, which means, ∑ky=1 Lt-CE(f(x), y) is not always a constant. However, as shown in Lemma 1, ∑ky=1 Lt-CE(f(x), y) is upper-bounded, and the bound gets tighter when t decreases. Specially, when t = 1, ∑ky=1 Lt-CE(f(x), y) = k - 1. In this case, Lt-CE becomes a symmetric loss, which can be considered equivalent to MAE. Based on Lemma 1, we further analyze the robustness of Lt-CE. We assume that the noisy example (x, ˜ y) is drawn from pη(x, ˜ y), and the ordinary example (x, y) is drawn from p(x, y). Note that this paper follows the most common setting where label noise is instance-independent. Then we have ˜ y = i (y = i) with probability ηi = (1 - η) and ˜ y = j with probability ηij for all j ̸= i and ∑j̸=i ηij = η. If ηij = η k-1 for all j ̸= i, then the noise is said to be uniform or symmetric, otherwise, the noise is said to be classconditional or asymmetric. Given any classifier f and loss function L, we define the risk of f under clean labels as RL(f) = Ep(x,y)[L(f(x), y)] and the risk under label noise rate η as RηL(f) = Epη(x,˜ y)[L(f(x), ˜ y)]. Let ˜ f and f ∗ be the global minimizers of RηLt-CE (f ) and RLt-CE (f ) respectively. Theorem 2. Under uniform label noise with η ≤ 1 - 1k , where Ct = ∑ti=1 1i is a constant that depends on t. Proof. Under uniform label noise, we have From Lemma 1 (by Eq. (6)), for all f, we have: where β = (1 - ηk k-1 ). On the other hand, we have: Thus, for ˜ f , where α = η(Ct - 1). Since η ≤ 1 - 1k , f ∗ is the global minimizer of RLt-CE (f) and ˜ f is the global minimizer of RηLt-CE (f ), we have which concludes the proof. Theorem 3. Under class-conditional label noise with ηij < 1 - ηi, ∀j ̸= i, ∀i, j ∈ [k], where ηij = p(˜ y = j|y = i), ∀j ̸= i and (1-ηi) = p(˜ y = i|y = i)), if RLt-CE (f ∗ ) = 0, then where A = (k - 1)(Ct - 1)Ep(x,y)(1 - ηi) > 0, Ct = ∑tn=1 1n and f ∗ is the global minimizer of RLt-CE (f) and ˜ f is the global minimizer of RηLt-CE (f). Proof. For class-conditional label noise, we have Then we have From our assumption that RLt-CE (f ∗ ) = 0, we have Lt-CE(f ∗ (x), y) = 0. This is only satisfied iff f ∗ j (x) = 1 when j = y and f ∗ j (x) = 0 when j ̸= y. According to the definition of Lt-CE, Lt-CE(f ∗ (x), j) = Ct, ∀j ̸= y and Lt-CE(f(x), j) ≤ Ct, ∀j ∈ [k]. Since f ∗ is the global minimizer of RLt-CE (f) and λ = (1 - ηi - ηij) > 0, we can obtain Therefore, we have which concludes the proof. Table 1: Average test accuracy (%) and standard deviation (over 5 trials) on benchmark datasets with symmetric label noise and asymmetric label noise. The best results are highlighted in bold. In addition, ·/◦ indicates whether the performance of our approach is statistically superior/inferior to the comparing approach on each dataset (paired t-test at 0.05 significance level). | Datasets | Methods | symmetric noise - 0.2 | symmetric noise - 0.4 | symmetric noise - 0.6 | symmetric noise - 0.8 | asymmetric noise - 0.1 | asymmetric noise - 0.2 | asymmetric noise - 0.3 | asymmetric noise - 0.4 | |------------|------------------------------------------------------------|---------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------|------------------------------------------------------------------------------------|------------------------------------------------------------------------------------|------------------------------------------------------------------------------------|------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------| | MNIST | CCE MAE MSE GCE SCE PHuber-CEτ=10 Taylor-CEt=2 | 98.38±0.16• 98.80±0.09• 98.86±0.06• 98.78±0.08• 98.93±0.21 98.46±0.08• 98.99±0.14 | 97.38±0.22• 98.32±0.08• 98.41±0.19 97.85±0.12• 98.44±0.21 97.60±0.11• 98.46±0.12 | 93.07±0.17• 97.15±0.07• 97.28±0.19 95.02±0.09• 96.99±0.21• 95.37±0.07• 97.53±0.23 | 90.48±0.76• 90.31±0.49• 90.14±0.19• 90.77±0.14• 90.01±0.21• 90.08±0.15• 91.14±0.18 | 98.35±0.09• 98.72±0.09• 98.44±0.11• 98.97±0.08 98.92±0.09• 98.65±0.07• 99.01±0.06 | 97.23±0.13• 88.77±0.12• 96.51±0.14• 98.61±0.11• 98.63±0.15• 96.97±0.11• 98.84±0.09 | 94.32±0.16• 88.13±0.18• 92.62±0.19• 96.95±0.16 97.74±0.21◦ 93.98±0.09• 96.69±0.13 | 79.67±0.17• 78.27±0.18• 86.22±0.19• 89.42±0.20◦ 85.97±0.19• 87.37±0.19• 87.68±0.23 | | Kuzushiji | CCE MAE MSE GCE SCE PHuber-CEτ=10 Taylor-CEt=2 | 82.92±0.15• 92.43±0.22• 90.59±0.18• 92.41±0.19• 93.03±0.14• 92.25±0.27• 93.34±0.16 | 79.75±0.39• 90.26±0.39 86.14±0.24• 91.48±0.17◦ 91.22±0.14◦ 88.07±0.23• 89.99±0.41 | 73.37±0.26• 82.61±0.21◦ 75.71±0.23• 87.46±0.28◦ 76.74±0.14• 76.56±0.31• 81.88±0.36 | 36.14±0.84• 60.46±0.54• 69.29±0.38◦ 66.70±0.34• 66.42±0.22• 68.45±0.25• 67.24±0.33 | 91.48±0.32• 94.11±0.21• 91.21±0.20• 94.18±0.16• 94.36±0.21• 93.61±0.26• 94.64±0.19 | 89.84±0.43• 93.18±0.29 89.29±0.56• 91.58±0.18• 92.57±0.22• 89.09±0.38• 93.34±0.23 | 85.75±1.06• 89.01±0.25◦ 83.96±0.74• 89.44±0.51◦ 88.82±0.28 84.65±0.84• 88.53±0.36 | 79.41±0.89• 82.43±0.67• 76.72±1.12• 79.15±0.44• 75.15±0.35• 77.94±1.36• 82.89±0.36 | | Fashion | CCE MAE MSE GCE SCE PHuber-CEτ=2 | 89.15±0.14• 89.68±0.12• 89.62±0.21• 89.81±0.08• 89.31±0.12• 89.56±0.21• | 88.04±0.15• 88.49±0.23• 88.25±0.36• 87.61±0.11• 87.61±0.28• 87.96±0.20• 88.97±0.28 | 79.15±0.21• 85.37±0.37• 86.79±0.35 86.75±0.29• 86.06±0.32• 85.27±0.26• 87.07±0.31 | 65.70±0.33• 70.02±0.42• 74.11±0.44• 78.39±0.64• 76.44±0.37• 77.75±0.28• 78.95±0.47 | 89.75±0.11• 90.01±0.09• 89.89±0.13• 89.76±0.11• 90.37±0.06◦ 90.17±0.12 90.25±0.09 | 82.91±0.21• 89.88±0.14• 89.38±0.12• 88.34±0.09• 90.26±0.10 89.68±0.11• 90.31±0.13 | 77.77±0.32• 89.09±0.23◦ 89.11±0.21◦ 86.21±0.19• 89.69±0.16◦ 88.55±0.24◦ | 72.45±0.59• 85.81±0.49• 86.53±0.52• 84.82±0.35• 85.82±0.34• 78.97±0.41• | | CIFAR-10 | Taylor-CEt=4 CCE MAE MSE GCE SCE PHuber-CEτ=2 Taylor-CEt=2 | 89.96±0.11 74.89±0.32• 85.53±1.02• 75.63±2.36• 86.04±0.11 84.12±0.08• 85.81±0.21 85.96±0.09 | 57.27±0.44• 79.28±0.88• 55.18±3.52• 75.72±0.09• 66.22±0.09• 80.25±0.22• 80.51±0.11 | 36.65±0.35• 65.70±0.76• 35.41±4.12• 48.34±0.12• 43.84±0.08• 67.71±0.19◦ 66.36±0.32 | 17.21±1.12• 31.07±4.13• 16.65±3.22• 18.92±0.21• 16.19±0.09• 32.97±0.32• 33.48±0.44 | 85.57±0.33• 80.38±0.67• 86.19±0.22• 87.77±0.07◦ 87.02±0.01• 87.91±0.13◦ 87.34±0.12 | 82.35±0.35• 79.06±0.58• 81.87±0.34• 83.43±0.07• 83.83±0.03• 84.87±0.26 85.02±0.11 | 88.31±0.15 78.14±0.55• 73.42±0.89• 76.85±0.32• 77.41±0.10• 77.67±0.04• 79.01±0.38• 79.37±0.12 | 87.38±0.25 72.02±0.51• 59.78±0.87• 72.38±0.39• 71.31±0.11• 72.51±0.14 71.87±0.36• 72.65±0.11 | | CIFAR-100 | CCE MAE MSE GCE SCE PHuber-CEτ=10 Taylor-CEt=6 | 47.00±0.13• 33.33±0.82• 47.66±0.63• 58.99±0.13 47.32±0.09• 58.11±0.11• 59.11±0.11 | 34.34±0.23• 26.56±0.79• 32.94±0.58• 50.37±0.14• 33.87±0.19• 50.89±0.13 50.99±0.09 | 19.37±0.33• 12.26±0.83• 18.41±0.73• 39.41±0.19◦ 18.79±0.26• 35.85±0.29• 38.31±0.12 | 7.34±0.21• 2.01±0.01• 7.59±0.79• 15.26±0.16• 7.28±0.38• 13.83±0.25• 15.96±0.31 | 56.71±0.34• 33.74±0.37• 56.02±0.47• 60.31±0.09• 56.84±0.12• 60.07±0.09• 60.96±0.21 | 50.02±0.62• 33.01±0.46• 48.63±0.45• 56.49±0.12◦ 49.82±0.16• 53.30±0.10• 55.45±0.12 | 43.29±0.58• 30.25±0.44• 40.69±0.52• 45.78±0.11 43.54±0.22• 44.39±0.14• 45.81±0.19 | 35.01±0.67• 22.66±0.53• 34.15±0.51• 34.99±0.14• 35.98±0.23◦ 35.36±0.13 35.45±0.25 | Table 2: Summary of benchmark datasets and models. | Dataset | # Train | # Test | # Feature | # Class | Model | |-----------|-----------|----------|-------------|-----------|-----------| | MNIST | 60000 | 10000 | 784 | 10 | LeNet-5 | | Fashion | 60000 | 10000 | 784 | 10 | LeNet-5 | | Kuzushiji | 60000 | 10000 | 784 | 10 | LeNet-5 | | CIFAR-10 | 50000 | 10000 | 3072 | 10 | ResNet-34 | | CIFAR-100 | 50000 | 10000 | 3072 | 100 | ResNet-34 | Theorem 2 and Theorem 3 show that using TCE, the difference of the risks caused by the derived hypotheses ˜ f and f ⋆ under noisy labels and clean labels are always bounded. Besides, the two bounds are related to the parameter t. Smaller t results in smaller Ct, hence both bounds in Theorem 2 and Theorem 3 would be tighter if t gets smaller. The above analysis clearly demonstrates the noise-tolerant ability of TCE. Specially when t = 1, TCE has the same theoretical guarantees as MAE. ## 4 Experiments ## 4.1 Experimental Settings Baselines. We compare our proposed TCE with CCE and multiple state-of-the-art robust loss functions. All the used loss functions in this paper are listed as follows. 1) CCE: The standard categoriccal cross entropy loss. It is neither symmetric nor bounded. 2) MAE: A symmetric loss function that has been demonstrated [Ghosh et al., 2017] to be robust to label noise. 3) MSE: The mean squared error. It is not symmetric but bounded. 4) GCE [Zhang and Sabuncu, 2018]: A bounded loss function that uses a hyper-parameter q to balance between MAE and CCE. The hyper-parameter q is set to 0.7, as this is the recommended setting by the corresponding paper [Zhang and Sabuncu, 2018]. 5) PHuber-CE: A Partially Huberised loss of CCE that corrects CCE on hard examples by gradient clipping. The hyper-parameter τ is selected from {2, 10}. 6) SCE [Wang et al., 2019]: The approach boosts CE symmetrically with a noise robust counterpart Reverse Cross Entropy (RCE). The regularization parameters α and β are configured with the suggested values on different datasets. 7) TCE: The loss function proposed in our paper, is based on the Taylor Series of CCE. The hyper-parameter t is selected from {2, . . . , 6}. For all the methods, learning rate is selected from {10-2 , 10-3 , 10-4 , 10-5 }. Datasets. Our experiments are conducted on MNIST [LeCun et al., 1998], Fashion-MNIST (Fashion in short) [Xiao et al., 2017], Kuzushiji-MNIST (Kuzushiji in short) [Clanuwat et al., 2018], CIFAR-10 [Krizhevsky et al., 2009] and CIFAR-100 [Krizhevsky et al., 2009] with two types of label noise: symmetric noise and asymmetric noise. We use appropriate networks to train different datasets, and all networks are trained using the Adam optimizer [Kingma and Ba, 2014] with the number of epochs set to 200 and the batch size set to 256. We test different noise rates, with η ∈ {20%, 40%, 60%, 80%} for symmetric label noise and η ∈ {10%, 20%, 30%, 40%} for asymmetric label noise. Figure 2: Test accuracy against number of epochs for training with 40% symmetric label noise. ![Image](https://ijcai.org/proceedings/2020/fulltext/305/image_000001_ab73399effbb9d3ded007b23b78bbcf26e6fd7a325cf01b24d63e879f9e7bc39.png) Figure 3: Test accuracy against number of epochs for training with 20% asymmetric label noise. ![Image](https://ijcai.org/proceedings/2020/fulltext/305/image_000002_3a3f8b03c8f90dc30336d63990d97ee553399ce58787f2aa554e2ea8cc3d9e71.png) Training Details. For MNIST/Fashion/Kuzushiji, LeNet-5 [LeCun et al., 1998] is used. Since they are MNIST-type datasets, noisy labels are generated in the same way for all of them. Symmetric label noise is generated by mapping a true label to a random label by a given probability η c-1 . For asymmetric label noise, flipping labels only occur within a specific set of classes [Patrini et al., 2017]. For MNIST, flipping 2 → 7, 3 → 8, 5 → 6 and 7 → 2. On the three datasets, networks are trained with weight decay of 10-4 . For CIFAR-10/CIFAR-100, ResNet-34 [He et al., 2016] is used. We perform 32 × 32 random crops after padding with 4 pixels on each side on the two datasets. Symmetric noise is generated in the same way as that for MNIST-type datasets. Following [Patrini et al., 2017], asymmetric label noise is generated by mapping TRUCK→AUTOMOBILE, BIRD→AIRPLANE, DEER→HORSE, and CAT↔ DOG with probability η for CIFAR-10. For CIFAR-100, the 100 classes are grouped into 20 super-classes with each has 5 sub-classes, and we flip between two randomly selected subclasses within each super-class. On the two datasets, networks are trained with weight decay of 0. ## 4.2 Experimental Results Table 1 reports the detailed experimental results of each loss function on the benchmark datasets. In Table 1, ·/◦ indicates whether the performance of our proposed approach is statistically superior/inferior to other comparing approaches on each dataset (paired t-test at 0.05 significance level). Out of the total 240 cases (with 6 comparing approaches, 5 datasets, and 8 label noise settings), our proposed approach is statistically superior to other comparing approaches in 83.33% cases and inferior to the comparing approaches in only 8.33% cases. Further more, in all cases, Lt-CE is superior to CE, MAE and MSE. As we showed above, Lt-CE is derived from CE, with MAE and MSE as average components. Thus, we may infer that Lt-CE maintains their advantages and surpasses them. ## 4.3 Parametric Analysis We also conduct experiments on MNIST, Fashion, Kuzushij, and CIFAR-10 with 40% symmetric label noise and 20% asymmetric label noise, for parametric analysis. We also include CCE for comparison, and vary the hyper-parameter t in {1, . . . , 6}. As shown in Figure 2 and Figure 3, CCE always leads to over-fitting and MAE cannot achieve a high classification accuracy in some cases. In contrast, Lt-CE avoids over-fitting and achieves good performance by varying t. Obviously, Lt-CE inherits the advantages of CCE while Lt-CE is more robust to label noise than CCE. ## 5 Conclusion In this paper, we propose a general framework dubbed Taylor cross entropy loss to train deep models in the presence of label noise. Our framework can not only enable to weight the extent of fitting the training labels by controlling the order of Taylor Series for Categorical Cross Entropy (CCE) loss, but also reveals the intrinsic relationships between CCE and other loss functions, such as Mean Absolute Error (MAE) and Mean Squared Error (MSE). Moreover, we present a detailed theoretical analysis to certify the robustness of this framework. Experiments on benchmark datasets also show that the proposed framework is superior to the state-of-the-art counterparts. In future work, we will explore if there exist robust loss functions that do not include any hyper-parameters. ## Acknowledgements This research is supported by NSOE-TSS2019-01, AISG-RP2019-0013, and NTU. ## References 1. [Berthon et al., 2020] Antonin Berthon, Bo Han, Gang Ni- u, Tongliang Liu, and Masashi Sugiyama. Confidence s- cores make instance-dependent label-noise learning possi- ble. arXiv preprint arXiv:2001.03772, 2020. 2. [Clanuwat et al., 2018] Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv preprint arXiv:1812.01718, 2018. 3. [Ghosh et al., 2015] Aritra Ghosh, Naresh Manwani, and P- S Sastry. Making risk minimization tolerant to label noise. Neurocomputing, 160:93-107, 2015. 4. [Ghosh et al., 2017] Aritra Ghosh, Himanshu Kumar, and P- S Sastry. Robust loss functions under label noise for deep neural networks. In AAAI, 2017. 5. [Goldberger and Ben-Reuven, 2017] Jacob Goldberger and Ehud Ben-Reuven. Training deep neural-networks using a noise adaptation layer. In ICLR, 2017. 6. [Han et al., 2018a] Bo Han, Gang Niu, Jiangchao Yao, X- ingrui Yu, Miao Xu, Ivor Tsang, and Masashi Sugiyama. Pumpout: A meta approach to robust deep learning with noisy labels. arXiv preprint arXiv:1809.11008, 2018. 7. [Han et al., 2018b] Bo Han, Jiangchao Yao, Gang Niu, Mingyuan Zhou, Ivor Tsang, Ya Zhang, and Masashi Sugiyama. Masking: A new perspective of noisy super- vision. In NuerIPS, pages 5836-5846, 2018. 8. [Han et al., 2018c] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NuerIPS, pages 8527-8537, 2018. 9. [He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016. 10. [Hendrycks et al., 2018] Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train deep networks on labels corrupted by severe noise. In NuerIPS, pages 10456-10465, 2018. 11. [Jiang et al., 2017] Lu Jiang, Zhengyuan Zhou, Thomas Le- ung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven curriculum for very deep neural networks on cor- rupted labels. arXiv preprint arXiv:1712.05055, 2017. 12. [Khetan et al., 2017] Ashish Khetan, Zachary C Lipton, and Anima Anandkumar. Learning from noisy singly-labeled data. arXiv preprint arXiv:1712.04577, 2017. 13. [Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. 14. [Krizhevsky et al., 2009] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny im- ages. Technical Report, 2009. 15. [LeCun et al., 1998] Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning ap- plied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998. 16. [Menon et al., 2015] Aditya Menon, Brendan Van Rooyen, Cheng Soon Ong, and Bob Williamson. Learning from corrupted binary labels via class-probability estimation. In ICML, pages 125-134, 2015. 17. [Menon et al., 2020] Aditya Krishna Menon, Ankit Singh Rawat, Sashank J. Reddi, and Sanjiv Kumar. Can gradient clipping mitigate label noise? In ICLR, 2020. 18. [Natarajan et al., 2013] Nagarajan Natarajan, Inderjit S D- hillon, Pradeep K Ravikumar, and Ambuj Tewari. Learn- ing with noisy labels. In NuerIPS, pages 1196-1204, 2013. 19. [Patrini et al., 2017] Giorgio Patrini, Alessandro Rozza, A- ditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In CVPR, pages 1944-1952, 2017. 20. [Sukhbaatar and Fergus, 2014] Sainbayar Sukhbaatar and Rob Fergus. Learning from noisy labels with deep neural networks. arXiv preprint arXiv:1406.2080, 2(3):4, 2014. 21. [Tanaka et al., 2018] Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. In CVPR, pages 5552-5560, 2018. 22. [Wang et al., 2019] Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In ICCV, pages 322-330, 2019. 23. [Wei et al., 2020] Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A join- t training method with co-regularization. arXiv preprint arXiv:2003.02752v3, 2020. 24. [Xia et al., 2019] Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learn- ing? In NuerIPS, pages 6835-6846, 2019. 25. [Xiao et al., 2017] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: A novel image dataset for bench- marking machine learning algorithms. arXiv preprint arX- iv:1708.07747, 2017. 26. [Yang et al., 2019] Hansi Yang, Quanming Yao, Bo Han, and Gang Niu. Searching to exploit memorization effec- t in learning from corrupted labels. arXiv preprint arX- iv:1911.02377, 2019. 27. [Yi and Wu, 2019] Kun Yi and Jianxin Wu. Probabilistic end-to-end noise correction for learning with noisy labels. In CVPR, pages 7017-7025, 2019. 28. [Yu et al., 2019] Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor W Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In ICML, pages 7164-7173, 2019. 29. [Zhang and Sabuncu, 2018] Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In NuerIPS, pages 8778-8788, 2018.