Can Cross Entropy Loss Be Robust to Label Noise?

1 Introduction

Deep Neural Networks (DNNs) have achieved great advances over the past years. With proper training, DNNs can easily achieve great classification performance. However, the success of DNNs relies on a large number of high-quality samples during the training process. Unfortunately, incorrect labels in large-scale datasets are often inevitable. In most scenarios, it can be more beneficial to have datasets with more but noisier labels than less but more accurate labels [Khetan et al., 2017]. Therefore, training a robust classifier in the presence of label noise is an increasingly valued task.

In general, (softmax) Categorical Cross Entropy (CCE) loss is the standard loss function used to train deep models. However, past studies [Ghosh et al., 2017; Patrini et al., 2017] show that using the standard CCE to train deep models leads to serious over-fitting (on noisy labels) and results in poor generalization ability. To mitigate this issue, increasing interests have been drawn in exploiting robust loss functions for training deep models against label noise. For example, the works of [Ghosh et al., 2015; Ghosh et al., 2017] reveal that symmetric loss functions, in which the sum of the risks over all categories is equivalent to a constant for each arbitrary example, can be robust to label noise. Representative symmetric loss functions include Ramp Loss [Ghosh et al., 2015] and (softmax) Mean Absolute Error (MAE) [Ghosh et al., 2017]. By both theoretical and empirical analysis, symmetric loss functions are demonstrated to be robust to label noise. Recently, several other loss functions that do not satisfy the symmetry condition strictly, including Generalized Cross Entropy (GCE) loss [Zhang and Sabuncu, 2018], Partial Huberised Cross Entropy (PHuberCE) loss [Menon et al., 2020] and Symmetric Cross Entropy (SCE) loss [Wang et al., 2019], have also been proposed to be robust against label noise when training deep neural networks. Although these loss functions achieve robustness to label noise in different ways, one thing in common is that these loss functions are derived from the standard CCE. In spite of their effectiveness, they all fail to embody the intrinsic relationships between CCE and other loss functions.

∗Corresponding author.

Motivated by the above observations, we wonder whether CCE can be further exploited to design a general framework that embody the intrinsic relationships between CCE and other loss functions, for robust learning with label noise. To answer this question, this paper proposes a general robust learning framework to train deep models in the presence of label noise. Specifically, we apply Taylor Series to derive an alternative representation of CCE. Moreover, we can flexibly adjust the order of Taylor Series to approximate CCE, and we call the approximated CCE Taylor Cross Entropy (TCE). In this framework, the order of Taylor Series reflects the extent of how TCE approximates to CCE. Furthermore, by varying the order of Taylor Series, we are able to reveal the intrinsic relationships between CCE and other loss functions. For example, MAE can be considered as the first-order Taylor Series approximation of CCE. The second-order Taylor Series approximation of CCE is an average combination of MAE and a lower bound of Mean Squared Error (MSE). The sufficiently large-order Taylor Series approximation of CCE recovers CCE. In addition, we show that our proposed TCE is upperbounded with finite order (which means the risk would never be infinite), and TCE is always an upper bound of MAE. In other words, minimizing TCE naturally enables to minimize MAE to some degree.

To sum up, our main contributions are three-fold:

• We propose a general robust learning framework dubbed Taylor Cross Entropy (TCE) loss. Taylor Series is applied to obtain a representation of CCE for training deep models in the presence of label noise.
• We present a detailed theoretical analysis to certify the robustness of TCE against label noise.
• We conduct extensive experiments for learning from symmetric and asymmetric label noise, and compare with a number of loss functions. Extensive experimental results on benchmark datasets demonstrate that our proposed approach significantly outperforms the stateof-the-art counterparts.

2 RelatedWork

In this section, we briefly review existing works on learning in the presence of label noise.

Noise rate estimation. Some of the early works [Natarajan et al., 2013; Sukhbaatar and Fergus, 2014; Menon et al., 2015; Patrini et al., 2017] aim to estimate the label transition matrix (sometimes called confusion matrix), and use it to train the target model. For this type of approach, the classification performance hinges on the quality of noise rate estimation [Goldberger and Ben-Reuven, 2017; Hendrycks et al., 2018; Han et al., 2018b; Xia et al., 2019]. However, noise rate estimation is challenging, especially on datasets with a huge number of classes.

Robust loss functions. Designing loss functions that are robust to label noise has been received increasing attention from researchers. The first work is from [Ghosh et al., 2015], which demonstrates that binary loss functions that satisfy the symmetric condition ℓ(z) + ℓ(-z) = c (e.g., ramp loss and sigmoid loss) where c is a constant, are robust to label noise for binary classification. Then, for multi-class classification, loss functions that satisfy the symmetric condition ∑kj=1 L(f(x), j) = C (e.g., MAE) where C is a constant, are demonstrated to be robust to label noise for the multiclass classification [Ghosh et al., 2017]. However, a recent study [Zhang and Sabuncu, 2018] shows that MAE is not able to achieve good performance on complicated datasets, due to its optimization issue. To alleviate this problem, Generalized Cross Entropy (GCE) [Zhang and Sabuncu, 2018] adopts the negative Box-Cox transformation strategy, and uses a hyperparameter q to balance between MAE and CCE. Partial Huberised Cross Entropy (PHuber-CE) [Menon et al., 2020] corrects CCE on hard examples by gradient clipping. Symmetric Cross Entropy (SCE) [Wang et al., 2019] combines CCE and Reverse Cross Entropy (RCE, which is equivalent to MAE) by tuning the regularization parameters. Although the above loss functions stem from CCE and some of them may be able to recover MAE or CCE by tuning the parameters, they all fail to embody the intrinsic relationships between CCE and other loss functions.

Other deep learning methods. There are some other approaches that adopt other solutions [Wei et al., 2020; Tanaka et al., 2018; Han et al., 2018a; Yu et al., 2019; Berthon et al., 2020; Yang et al., 2019] to deal with noisy labels. For example, MentorNet [Jiang et al., 2017] is trained to supervise the training of a StudentNet with a sample weighting scheme. Co-teaching [Han et al., 2018c] trains two networks simultaneously and enables the two networks to learn from each other. PENCIL [Yi and Wu, 2019] trains neural networks using label probability distributions and updates these distributions in each epoch.

3 Taylor Cross Entropy Loss for Robust Learning with Label Noise

In this section, we first briefly review CCE and MAE. Then, we introduce our proposed Taylor cross entropy loss. Finally, we theoretically analyze the robustness of Taylor cross entropy loss.

3.1 Preliminaries

We consider the problem of k-class classification. Suppose the clean data set is represented as D = {(xi, yi) | 1 ≤ i ≤ m}, where xi ∈ X (X ∈ Rd ) is a d-dimensional feature vector and yi ∈ {1, . . . , k} is the label associated with xi. A classifier is a function that maps the feature space to the label space f : X → Rk . In this paper, we consider the common case where the function f is a DNN with the softmax output layer. In this way, the commonly used loss functions CCE and MAE can be represented as:

where the fy(x) denotes the y-th element of f(x) and ey is a one-hot vector with eyj = 1 if j = y, otherwise 0. Besides, the gradients of CCE and MAE can be shown as:

where θ is the set of parameters of f. As shown in the above equations, when training with CCE, examples with smaller prediction confidences are weighted more than examples with larger prediction confidences for gradient update. In other words, CCE pays more attention to hard examples. On the contrary, MAE treats all the examples equally. Hence CCE is more preferred than MAE when training with clean data. However, in the presence of label noise, the performance of CCE dramatically drops, since the given labels of hard examples may be incorrect. In the extreme case, the weight becomes infinite if fy(x) → 0, which leads to serious overfitting on noisy labels. In contrast, MAE does not suffer from this problem, since each example is treated equally. Therefore, there arises a question: can cross entropy loss be robust to label noise? To answer this question, we propose a general framework of CCE that endows the robustness to CCE against label noise.

3.2 Taylor Cross Entropy Loss

Given a function g(x), if g(x) is differentiable to order n at x = x0, then g(x) can be written as a Taylor Series:

where gi (x0) denotes the i-th order derivative of g(x) at x0. Recall that LCCE(f(x), y) = -log fy(x), we can define g(fy(x)) = - log fy(x). Then we have

If we set fy(x0) = 1, then ∀i ≥ 1 we have

Thus we can express LCCE as

Obviously, it is unrealistic to take into account all the terms, since n → ∞. Hence we propose to reserve finite terms in Eq. (3) and obtain an approximation of CCE, which is called Taylor Cross Entropy (TCE) loss:

where t ∈ N+ is a hyper-parameter that denotes the order of the Taylor Series, in other words, the proximity to the CCE. Figure 1 illustrates Lt-CE with t varying in {1, . . . , 10}. According to the definition of Lt-CE and Figure 1, we can obtain some interesting deductions.

Theorem 1. Our proposed Taylor cross entropy loss has the following properties: 1) When t = 1, Lt-CE = 1 2 LMAE; 2) When t = 2, Lt-CE is an average combination of LMAE and a lower bound of LMSE; 3) When t →∞, Lt-CE is equivalent to LCCE; 4) LMAE ≤ 2Lt-CE, ∀t ∈ N+.

Proof. 1) When t = 1, Lt-CE is represented as

It is clear that in this case, Lt-CE = 12 LMAE, which means, minimizing Lt-CE is equivalent to minimizing LMAE.

1. When t = 2, Lt-CE is represented as

For the first term, we have already shown (1 - fy(x)) = 12 LMAE. For the second term, we have

Hence the second term is a lower bound of MSE.

1. According to the definition of Lt-CE, it is clear that Lt-CE will be equivalent to LCCE if t →∞.
2. Based on Property 1), we have

Since the second term is always non-negative, LMAE ≤ 2Lt-CE(f(x), y), and the equality holds when and only when t = 1. However, the difference becomes larger as t increases, which also suggests that LCCE is not robust from another point of view, since LMAE is the standard robust loss with strong theoretic guarantees [Ghosh et al., 2017].

Properties 1) and 3) show that LMAE and LCE can be considered as special cases of Lt-CE. Properties 2) and 4) certify the robustness of Lt-CE to some degree, since LMAE and LMSE have been shown to be robust in the presence of label noise [Ghosh et al., 2017].

3.3 Theoretical Analysis

Here, we theoretically analyze the robustness of our proposed Taylor cross entropy loss.

By the definition of Lt-CE (i.e., Eq. (4)), we can easily derive an upper bound and a lower bound of Lt-CE as follows:

Lemma 1. For any x and any positive integer t < +∞, the sum of Lt-CE with respect to all the classes satisfies:

where Ct = ∑ti=1 1i is a constant that depends on t.

Proof. Based on Eq. (5), if we consider the sum of Lt-CE with respect to all the classes, the following equality holds:

Hence

which concludes the proof.

We can find that Lt-CE is not always a symmetric loss, which means, ∑ky=1 Lt-CE(f(x), y) is not always a constant. However, as shown in Lemma 1, ∑ky=1 Lt-CE(f(x), y) is upper-bounded, and the bound gets tighter when t decreases. Specially, when t = 1, ∑ky=1 Lt-CE(f(x), y) = k - 1. In this case, Lt-CE becomes a symmetric loss, which can be considered equivalent to MAE.

Based on Lemma 1, we further analyze the robustness of Lt-CE. We assume that the noisy example (x, ˜ y) is drawn from pη(x, ˜ y), and the ordinary example (x, y) is drawn from p(x, y). Note that this paper follows the most common setting where label noise is instance-independent. Then we have ˜ y = i (y = i) with probability ηi = (1 - η) and ˜ y = j with probability ηij for all j ̸= i and ∑j̸=i ηij = η. If ηij = η k-1 for all j ̸= i, then the noise is said to be uniform or symmetric, otherwise, the noise is said to be classconditional or asymmetric. Given any classifier f and loss function L, we define the risk of f under clean labels as RL(f) = Ep(x,y)[L(f(x), y)] and the risk under label noise rate η as RηL(f) = Epη(x,˜ y)[L(f(x), ˜ y)]. Let ˜ f and f ∗ be the global minimizers of RηLt-CE (f ) and RLt-CE (f ) respectively.

Theorem 2. Under uniform label noise with η ≤ 1 - 1k ,

where Ct = ∑ti=1 1i is a constant that depends on t.

Proof. Under uniform label noise, we have

From Lemma 1 (by Eq. (6)), for all f, we have:

where β = (1 - ηk k-1 ). On the other hand, we have:

Thus, for ˜ f ,

where α = η(Ct - 1). Since η ≤ 1 - 1k , f ∗ is the global minimizer of RLt-CE (f) and ˜ f is the global minimizer of RηLt-CE (f ), we have

which concludes the proof.

Theorem 3. Under class-conditional label noise with ηij < 1 - ηi, ∀j ̸= i, ∀i, j ∈ [k], where ηij = p(˜ y = j|y = i), ∀j ̸= i and (1-ηi) = p(˜ y = i|y = i)), if RLt-CE (f ∗ ) = 0, then

where A = (k - 1)(Ct - 1)Ep(x,y)(1 - ηi) > 0, Ct = ∑tn=1 1n and f ∗ is the global minimizer of RLt-CE (f) and ˜ f is the global minimizer of RηLt-CE (f).

Proof. For class-conditional label noise, we have

Then we have

From our assumption that RLt-CE (f ∗ ) = 0, we have Lt-CE(f ∗ (x), y) = 0. This is only satisfied iff f ∗ j (x) = 1 when j = y and f ∗ j (x) = 0 when j ̸= y. According to the definition of Lt-CE, Lt-CE(f ∗ (x), j) = Ct, ∀j ̸= y and Lt-CE(f(x), j) ≤ Ct, ∀j ∈ [k]. Since f ∗ is the global minimizer of RLt-CE (f) and λ = (1 - ηi - ηij) > 0, we can obtain

Therefore, we have

which concludes the proof.

Theorem 2 and Theorem 3 show that using TCE, the difference of the risks caused by the derived hypotheses ˜ f and f ⋆ under noisy labels and clean labels are always bounded. Besides, the two bounds are related to the parameter t. Smaller t results in smaller Ct, hence both bounds in Theorem 2 and Theorem 3 would be tighter if t gets smaller. The above analysis clearly demonstrates the noise-tolerant ability of TCE. Specially when t = 1, TCE has the same theoretical guarantees as MAE.

4 Experiments

4.1 Experimental Settings

Baselines. We compare our proposed TCE with CCE and multiple state-of-the-art robust loss functions. All the used loss functions in this paper are listed as follows. 1) CCE: The standard categoriccal cross entropy loss. It is neither symmetric nor bounded. 2) MAE: A symmetric loss function that has been demonstrated [Ghosh et al., 2017] to be robust to label noise. 3) MSE: The mean squared error. It is not symmetric but bounded. 4) GCE [Zhang and Sabuncu, 2018]: A bounded loss function that uses a hyper-parameter q to balance between MAE and CCE. The hyper-parameter q is set to 0.7, as this is the recommended setting by the corresponding paper [Zhang and Sabuncu, 2018]. 5) PHuber-CE: A Partially Huberised loss of CCE that corrects CCE on hard examples by gradient clipping. The hyper-parameter τ is selected from {2, 10}. 6) SCE [Wang et al., 2019]: The approach boosts CE symmetrically with a noise robust counterpart Reverse Cross Entropy (RCE). The regularization parameters α and β are configured with the suggested values on different datasets. 7) TCE: The loss function proposed in our paper, is based on the Taylor Series of CCE. The hyper-parameter t is selected from {2, . . . , 6}. For all the methods, learning rate is selected from {10-2 , 10-3 , 10-4 , 10-5 }.

Datasets. Our experiments are conducted on MNIST [LeCun et al., 1998], Fashion-MNIST (Fashion in short) [Xiao et al., 2017], Kuzushiji-MNIST (Kuzushiji in short) [Clanuwat et al., 2018], CIFAR-10 [Krizhevsky et al., 2009] and CIFAR-100 [Krizhevsky et al., 2009] with two types of label noise: symmetric noise and asymmetric noise. We use appropriate networks to train different datasets, and all networks are trained using the Adam optimizer [Kingma and Ba, 2014] with the number of epochs set to 200 and the batch size set to 256. We test different noise rates, with η ∈ {20%, 40%, 60%, 80%} for symmetric label noise and η ∈ {10%, 20%, 30%, 40%} for asymmetric label noise.

Training Details. For MNIST/Fashion/Kuzushiji, LeNet-5 [LeCun et al., 1998] is used. Since they are MNIST-type datasets, noisy labels are generated in the same way for all of them. Symmetric label noise is generated by mapping a true label to a random label by a given probability η c-1 . For asymmetric label noise, flipping labels only occur within a specific set of classes [Patrini et al., 2017]. For MNIST, flipping 2 → 7, 3 → 8, 5 → 6 and 7 → 2. On the three datasets, networks are trained with weight decay of 10-4 . For CIFAR-10/CIFAR-100, ResNet-34 [He et al., 2016] is used. We perform 32 × 32 random crops after padding with 4 pixels on each side on the two datasets. Symmetric noise is generated in the same way as that for MNIST-type datasets. Following [Patrini et al., 2017], asymmetric label noise is generated by mapping TRUCK→AUTOMOBILE, BIRD→AIRPLANE, DEER→HORSE, and CAT↔ DOG with probability η for CIFAR-10. For CIFAR-100, the 100 classes are grouped into 20 super-classes with each has 5 sub-classes, and we flip between two randomly selected subclasses within each super-class. On the two datasets, networks are trained with weight decay of 0.

4.2 Experimental Results

Table 1 reports the detailed experimental results of each loss function on the benchmark datasets. In Table 1, ·/◦ indicates whether the performance of our proposed approach is statistically superior/inferior to other comparing approaches on each dataset (paired t-test at 0.05 significance level). Out of the total 240 cases (with 6 comparing approaches, 5 datasets, and 8 label noise settings), our proposed approach is statistically superior to other comparing approaches in 83.33% cases and inferior to the comparing approaches in only 8.33% cases. Further more, in all cases, Lt-CE is superior to CE, MAE and MSE. As we showed above, Lt-CE is derived from CE, with MAE and MSE as average components. Thus, we may infer that Lt-CE maintains their advantages and surpasses them.

4.3 Parametric Analysis

We also conduct experiments on MNIST, Fashion, Kuzushij, and CIFAR-10 with 40% symmetric label noise and 20% asymmetric label noise, for parametric analysis. We also include CCE for comparison, and vary the hyper-parameter t in {1, . . . , 6}. As shown in Figure 2 and Figure 3, CCE always leads to over-fitting and MAE cannot achieve a high classification accuracy in some cases. In contrast, Lt-CE avoids over-fitting and achieves good performance by varying t. Obviously, Lt-CE inherits the advantages of CCE while Lt-CE is more robust to label noise than CCE.

5 Conclusion

In this paper, we propose a general framework dubbed Taylor cross entropy loss to train deep models in the presence of label noise. Our framework can not only enable to weight the extent of fitting the training labels by controlling the order of Taylor Series for Categorical Cross Entropy (CCE) loss, but also reveals the intrinsic relationships between CCE and other loss functions, such as Mean Absolute Error (MAE) and Mean Squared Error (MSE). Moreover, we present a detailed theoretical analysis to certify the robustness of this framework. Experiments on benchmark datasets also show that the proposed framework is superior to the state-of-the-art counterparts. In future work, we will explore if there exist robust loss functions that do not include any hyper-parameters.

Acknowledgements

This research is supported by NSOE-TSS2019-01, AISG-RP2019-0013, and NTU.