Interpreting and Evaluating Neural Network Robustness

1 Introduction

In the past few years, Neural Networks (NNs) have achieved superiors success in various domains, e.g., computer vision [Szegedy et al., 2016], speech recognition [Hinton et al., 2012], autonomous systems [Huval et al., 2015], etc. However, the recent appearance of adversarial attacks [Kurakin et al., 2016] greatly challenges the security of neural network applications: by crafting and injecting human-imperceptible noises into test inputs, neural networks' prediction results can be arbitrarily manipulated [Ian J. Goodfellow, 2014]. Until now, the emerging pace, effectiveness, and efficiency of new attacks always take an early lead to the defense solutions [Carlini and Wagner, 2017], and the key factors of the adversarial vulnerabilities are still unclear, leaving the neural network robustness study in a vicious cycle.

In this work, we aim to qualitatively interpret neural network models' adversarial vulnerability and robustness, and establish a quantitative metric for the model-intrinsic robustness evaluation. To interpret the robustness, we adopt the loss visualization technique [Goodfellow et al., 2015], which was widely used in model convergence studies. As adversarial attacks leverage perturbations in inputs, we switch the loss visualization from its original parameter space into the input space and illustrate how a neural network is deceived by adversarial perturbations. Based on the interpretation, we design a robustness evaluation metric to measure a neural network's maximum prediction divergence within a constrained perturbation range. We further optimize the metric evaluation process to keep its consistency under extrinsic factor variance, e.g., model reparameterization [Dinh et al., 2017].

Specifically, we have the following contributions:

• We interpret the adversarial vulnerability and robustness by defining and visualizing a new loss surface called decision surface. Compared to the cross-entropy based loss surface, the decision surface contains the implicit decision boundary and provides better visualization effect;
• We testify that adversarial deception is caused by the neural network's neighborhood under-fitting. Our visualization shows that adversarial examples are naturallyexisted points lying in the close neighborhood of the inputs. However, the neural network fails to classify them, which caused the adversarial example phenomenon;
• We propose a robustness evaluation metric. Combined with a new normalization method, the metric can invariantly reflect a neural network's intrinsic robustness property regardless of attacks and defenses;
• We reveal that under certain cases, e.g., defensive distillation, the commonly-used PGD adversarial testing accuracy can give unreliable robustness estimation, while our metric could reflect the model robustness correctly.

Extensive evaluation results show that our defined robustness metric could well indicate the model-intrinsic robustness across different datasets, various architectures, multiple adversarial attacks, and different defense methods.

2 Background and RelatedWork

2.1 Adversarial Attacks and Robustness

Adversarial examples were firstly introduced by [Szegedy et al., 2013], which revealed neural networks' vulnerability to adversarial noises and demonstrated the gap between the artificial cognition and human visual perception. Since then, various adversarial attacks were proposed, such as L-BFGS attack [Kurakin et al., 2016], FGSM attack [Ian J. Goodfellow, 2014], C&W attack [Carlini and Wagner, 2017], blackbox attack [Papernot et al., 2017], etc.

Driven by the appearance of adversarial attacks, corresponding defense techniques also emerged, including adversarial training [Ian J. Goodfellow, 2014], defensive distillation [Papernot et al., 2016], gradient regularization [Ross and Doshi-Velez, 2018], adversarial logit pairing [Kannan et al., 2018], etc. Among those, MinMax robustness optimization [Madry et al., 2018] is considered as one of the most potent defenses, which boosts model accuracy by integrating the worst-case adversarial examples into the model training.

Currently, testing accuracy under adversarial attacks is used to evaluate the model robustness. However, it is highly affected by the attack specifications and can't comprehensively reflect the actual robustness regarding model-intrinsic properties. For example, one commonly used way to evaluate the model robustness is adopting the testing accuracy under projected gradient descent (PGD) attack as an estimation. However, our experiments demonstrate that such a robustness estimation is highly unreliable: a model with a high PGD testing accuracy could be easily broken by other attacks.

In this work, we aim to provide an intrinsic robustness property evaluation metric that is invariant from the specifications of models, attacks, and defenses.

2.2 Neural Network Loss Visualization

Neural network loss visualization is considered as one of the most useful approaches in neural network analysis own to its intuitive interpretation. Proposed by [Goodfellow et al., 2015], the loss visualization is utilized to analyze model training and convergence. Later, [Keskar et al., 2017] further revealed that flat local minima is the key to model generalization in parameter space. However, a model reparameterization issue was identified by [Dinh et al., 2017] that the model parameter scaling may distort the geometry properties.

In this work, we adopt the concept of the loss visualization to analyze the neural network's loss behaviors under adversarial perturbations. Meanwhile, we will also provide a normalization method to solve the model reparameterization problem and derive our scaling-invariant robustness metric.

2.3 Visualization Space Selection

Besides of solving the reparameterization issue, the loss visualization needs further customization for the adversarial perturbation analysis. As the loss visualization mainly evaluates a neural network's generalization ability, it focuses on the parameter space to analyze the model training and convergence in previous works. However, such an analysis focus doesn't fit well in the adversarial attacks and defenses, whose action scope is in input space. On the other hand, loss function in the input space measures the network's loss variations w.r.t the input perturbations. It naturally shows the influence of adversarial perturbations and is suitable for studying the robustness to adversarial perturbations. Therefore, we extend the previous methods into the input space.

Figure 1: ResNet's loss surface in (a) Parameter space (b) Input space. The loss surface demonstrates significant non-smooth variation in input space, demonstrating its high sensitivity to noises.

Figure 1 showed two examples of the visualized loss surface of an ResNet model in the parameter space and the input space, which illustrate the difference between the two visualization spaces. Although the loss surface in the parameter space can show a flat minima, its significant non-smooth variations in the input space demonstrate the loss is highly sensitive to input perturbations, which can be adversarial vulnerabilities. In this work, we will adopt the input space as the default visualization setting for robustness interpretation.

3 Adversarial Robustness Interpretation

3.1 Neural Network Loss Visualization

Loss Visualization Basis

The prediction of a neural network can be evaluated by its loss function F(θ, x), where θ is the model parameter set (weight and bias) and x is the input. As the inputs x are usually constructed in a high-dimensional space, direct visualization analysis on the loss surface is impossible. To solve this issue, the loss visualization projects the high-dimensional loss surface into a low-dimensional space to visualize it (e.g. a 2D hyper-plane). During the projection, two vectors α and β are selected and normalized as the base vectors for x-y hyperplane. Given an starting input point o, the points around it can be interpolated, and the corresponding loss values can be

where, the original point o in the function F denotes the original image, α and β can be treated as the unit perturbation added into the image, and the coordinate (i, j) denotes the perturbation intensity. In the loss visualization, a point's coordinate also denotes its divergence degree from the original point along α and β direction. After sampling sufficient points' loss values, the function F with high-dimensional inputs could be projected to the chosen hyper-plane.

Decision Surface Construction

As the loss visualization is mostly used to analyze model convergence, the loss function F(θ, x) is usually represented by the cross-entropy loss, which constructs a conventional loss surface in the visualization. However, one critical limitation of the cross-entropy based loss surface is that, it cannot qualitatively show the explicit decision boundary of one input test, and less helpful for adversarial deception analysis.

Therefore, we propose a decision surface to replace the loss surface in the loss visualization:

where, Z(x) is the logit output before the softmax layer, and t is the true class index of the input x. The decision function S(x) evaluates the confidence of prediction. In the correct prediction cases, S(x) should always be positive, while S(x) < 0 denotes a wrong prediction. Specifically, S(x) = 0 indicates the equal confidence for both correct and wrong prediction, which is the decision boundary of model. Consequently, the visualization surface constructed by the function S(x) is defined the decision. Different from the cross-entropy based loss surface, the decision surface demonstrates explicit decision boundaries, and assist the adversarial analysis.

3.2 Visualizing Adversarial Vulnerability

Experimental Analysis

Based on the loss visualization, we project a neural network's loss behavior into 2D hyper-planes. By comparing the model's 4 different types loss behavior in decision surface, we provide a experimental analysis for the adversarial vulnerability.

As shown in Figure 2, the visualized hyper-planes have the central points as the original neural network inputs, and their x-axes share the same random input divergence direction - α. Meanwhile, each hyper-plane has a dedicated input divergence direction - β along the y-axis, which indicate 4 kinds of perturbations, including random noise, cross-entropy based non-targeted FGSM attack [Kurakin et al., 2016], Leastlikely targeted FGSM attack [Kurakin et al., 2016], and nontargeted C&W attack [Carlini and Wagner, 2017]. Specifically β values in the three adversarial attacks can be determined as:

where N is normal distribution, Z is the logit output, yt is the true class label, yl is least likely class label (both one-hot).

In Figure 2, we use arrows to show the shortest distance to cross the decision boundary L(x)=0. As projected in Figure 2(a), when the input is diverged by the perturbation along a random direction, it will take much longer distance to cross the decision boundary. This explains the common sense that natural images with small random noises won't degrade neural network accuracy significantly. By contrast, for the adversarial attacks projected in Figure 2(b)∼(d), the attacks find aggressive directions (β direction shown in y-axis), towards which the decision boundary is in the close neighborhood around the original input. Therefore, adding those small perturbations that even human can't perceive into input can mislead the model decision and generates adversarial examples.

Vulnerability Interpretation

The above experimental analysis reveals the nature of adversarial examples: Although a neural network seems to converge well after the model training (the demonstrated model achieves 90% accuracy on CIFAR10), there still exist large regions of image points that the neural network fails to classify correctly (as shown by the large regions beyond the decision boundary in Figure 2(b)∼(d)). What's worse, some of these regions are extremely close to the original input point (even within ℓinf < 1 distance).

Base on these analysis, we could conclude that, rather than being "generated" by attackers, the adversarial examples are "naturally existed" already that models fail to learn correctly. To fix such intrinsic vulnerability of neural networks, the essential and ultimate robustness enhancement should focus on solving the "neighborhood under-fitting" issue.

3.3 Interpreting Adversarial Robustness

To verify our geometric robustness theory, we compare two pairs of robust and natural models trained on MNIST and CIFAR10 respectively. These models are released from the adversarial attacking challenges 12 , and built with the same structure but different robustness degrees (natural training and MinMax training [Madry et al., 2018]).

To verify our theory, we visualize the models' decision surfaces for interpretation: (1) As shown in Figure 3, dramatic differences between the natural and robust decision surfaces can be observed: Natural (vulnerable) model's decision surfaces show sharp peaks and large slopes, where the decision confidence could quickly drop to negative areas (wrong classification regions). (2) By comparison, on robust decision surfaces (shown in Figure 3(c)(d)), all neighborhood points around the original input point are located on a high plateau with L(x) > 0 (correct classification regions). (3) The surface in the neighborhood is rather flat with negligible slopes until it reaches approximately ℓ∞ = 0.3 constraints, which is exactly the adversarial attack constraint used in robust training. Similar phenomenon could be observed in Figure 4 on CIFAR10.

Such robust model's loss geometry verifies our previous conclusion that, fixing the neighborhood under-fitting issue is the essential robustness enhancement solution for neural networks. And a flat and wide plateau around the original point on decision surface is one of the most desired properties of a robust model.

4 Adversarial Robustness Evaluation

4.1 Formal Definition of Robustness Metric

As aforementioned, the decision surface of a robust model should have a flat neighborhood around the input point x. Intuitively explanation is that a robust model should have good prediction stability-its prediction does not have significant change with small perturbations. In fact, models are not always robust-the predictions of a model on clean and noisy inputs are not always the same and can diverge to a large extent with small adversarial noises. As such, given a feasible perturbation set, the maximum divergence between the original prediction and the worst-case adversarial prediction could be used to denote the model's vulnerability degree (i.e., the inverses of model robustness).

1 https://github.com/MadryLab/mnist challenge

2 https://github.com/MadryLab/cifar10 challenge

Based on this definition, firstly, we calculate the divergence between two predictions on an original input and an adversarial input with perturbations in a defined range. Specifically, we use the Kullback-Leibler divergence, which is known as KL Divergence (DKL) and is a common evaluation metric on measuring the divergence between two probability distributions. The formal robustness could be estimated by:

where P (·) is the prediction results from the evaluated model. A lower divergence DKL indicates the model is more robust as a more stable prediction is maintained. The final robustness metric ψ(x) is defined inversely proportional to the maximumDKL since the largest divergence will generate the smallest robustness score ψ(x). To obtain the max term in Eq. 4, we use the gradient ascent algorithm to directly optimize the KL-divergence, which demonstrates accurate and stable estimations that we will show in Sec. 5.

4.2 Invariant Normalization against Model Reparameterization

The robustness metric defined in previous works has a problem called "model re-parameterization": on the condition that weights and biases are enlarged by the same coefficients simultaneously, a neural network model's prediction results and its robustness property will not change, while the defined KL divergence can have dramatic change [Dinh et al., 2017].

To solve this problem, we design a simple but effective normalization method: the basic idea is to add a scale-invariant normalization layer after the logit layer output. Since the neural network before the logit layer is piecewise-linear, we could then use normalization to safely remove the scaling effect of model reparameterization. The basic process is as follows: firstly, we attain a confidence vector of the logit layer, which can contain either positive or negative values; then we divide them by the max-absolute-value to normalize the confidence vector to the range of (-1, 1) and re-center them into positive range (0, 2). Owning to the max division, the final confidence vector will not change even when the parameters are linearly scaled up (or down). Finally, we use a simple sum-normalization to transform the confidence vector to a valid probability distribution. The overall normalization is:

Here P(x) is the final normalized probability distribution, ˜ F is the normalized confidence vector, F(x) is the original logit layer output, and x is the input. By the above normalization method, we could successfully alleviate the model reparameterization effect, which is shown in Sec. 5.

5 Robustness Evaluation Experiments

5.1 Experiment Setup

To test the generality of our metric for neural networks' robustness evaluation, we adopt three common datasets (i.e. MNIST, CIFAR10, and ImageNet) and different models for the experiment, including FcNet, LeNet, ConvNet, ResNet18, ResNet152, and DenseNet.

To further test our metric on neural networks with different robustness degrees, the following defense settings are applied: No Defense, Adversarial Training [Kurakin et al., 2016], Gradient Regularization Training [Ross and DoshiVelez, 2018], Defensive Distillation [Papernot et al., 2016], Gradient Inhibition [Liu et al., 2018] and MinMax Training [Madry et al., 2018] 3 .

Correspondingly, the robustness verification is based on referencing the adversarial testing accuracies from two currently strongest attacks: 30-step PGD (PGD-30) attack based on cross-entropy loss and 30-step CW (CW-30) attacks based on C&W loss. The adversarial perturbations are constrained by the ℓ∞-norm as 0.3/1.0, 8.0/255.0, 16.0/255.0 on MNIST, CIFAR10, and ImageNet respectively.

5.2 Robustness Metric Evaluation

MNIST Experiments. On MNIST dataset, the results are shown in Table 1: (1) The results firstly demonstrate that our metric could well reflect different robustness degrees on the same neural network model. For example, three FcNet models show increasing robustness in ψ(x), which aligns well with their reference accuracies from both PGD-30 and CW30 attack; (2) The results also show the generality of our metric on FcNet and LeNet models.

3 The gradient regularization and MinMax training is reimplemented with Pytorch, which may cause small deviations from the original reported accuracy.

*AdvTrain: [Kurakin et al., 2016], MinMax: [Madry et al., 2018].

Table 1: Robustness Metric Evaluation on MNIST

CIFAR10 Experiments. Table 2 shows the experimental results on CIFAR10, including three common neural network models (i.e. ConvNet, ResNet18, and DenseNet), as well as three robustness settings (i.e. No defense, Gradient Regularization, and MinMax Training). The experiment results show that our metric has the same scale with the referenced adversarial testing accuracies, implying our metric's good generality on complex neural network models and different defenses. To better illustrate a neural network model's robustness, we visualized three ResNet18 models with different robustness degrees in Figure 5. As the robustness degree increases, the models' loss surfaces become more and more smooth. Our empirical visualization results imply that the smoother decision surface in the input space indicates better adversarial robustness, which coincidentally matches the parameter space generalization hypothesis [Keskar et al., 2017].

ImageNet Experiments. In the experiment onMNIST and CIFAR10, our proposed robustness metric aligns well with adversarial testing accuracies of PGD-30 and CW-30. However, when we evaluate the MinMax model on ImageNet, two reference accuracies demonstrate certain inconsistency: The MinMax model is released as the base model of the state-ofthe-art defense on ImageNet by [Xie et al., 2018]. To conduct the MinMax training, the reported time needed is about 52 hours on 128 V100 GPUs. Despite that, the reported accuracy showed very good robustness estimation of the model, which can achieve 42.6% under 2000-iteration PGD attacks. However, when we more thoroughly evaluated the model by CW-30 attack, we found the model's testing accuracy is only 12.5% under the attack. We call such a case as "unreliable estimation" in PGD-based adversarial testing, whose robustness estimation cannot generalize to all attacks. We will discuss this case and several other similar ones in details in Sec. 5.3, and reveal the current deficiency of adversarial testing based robustness estimation.

*GradReg: [Ross and Doshi-Velez, 2018], MinMax: [Madry et al., 2018].

Table 2: Robustness Metric Evaluation on CIFAR10

5.3 Our Metric vs. Adversarial Testing Accuracy

As mentioned above, the adversarial testing accuracy from different adversarial attacks may demonstrate certain inconsistency, and therefore mislead the robustness estimation. In addition to the ImageNet example, we also include another two cases that the adversarial testing accuracies yield unreliable robustness estimation: defensive distillation [Papernot et al., 2016] and gradient inhibition [Liu et al., 2018].

To demonstrate the unreliability of these cases, we train three new models on MNIST and CIFAR10 respectively, using natural training, defensive distillation, and gradient inhibition methods. For the ImageNet model, we use a public released model 4 , which can achieve a state-of-the-art accuracy 45.5% against PGD-30 attack (within ℓ∞ ≤ 16/255).

The overall experimental results are shown in Table. 3, which shows that though all these defenses can achieve high PGD-30 adversarial testing accuracy, they actually bring very limited robustness improvement:

On MNIST and CIFAR10, the distillation and gradient inhibition defenses provide the models with high adversarial testing accuracy against both FGSM and PGD-30 attacks (even higher than state-of-the-art MinMax methods), which seemly indicates these models are significantly robust. However, when measured by our metric, we have the opposite conclusion: these models are merely as robust as no-defense models and incomparable to the robust models trained by MinMax. To further verify this conclusion, we test these models with more adversarial settings and the testing accuracy dramatically degrades to almost zero in all the tests.

The tests above further prove our statement: the adversarial testing accuracy based on PGD-30 may yield unreliable robustness estimation, which cannot reflect the model's intrinsic robustness. This is because the distillation and gradient inhibition both rely on the input gradient vanishing to achieve robustness enhancement, which is mainly provided by the nonlinear softmax and negative log loss. Since C&W attack doesn't rely on the cross-entropy loss, it can easily crack those two defenses. Such a case also applies to the ImageNet model trained with MinMax defenses as shown in the last two rows of Table. 3.

In contrast, our robustness metric can successfully reflect

4 MinMax model is obtained in following link: https://github.com /facebookresearch/imagenet-adversarial-training.

*Distillation: [Papernot et al., 2016], GradInhib: [Liu et al., 2018], MinMax: [Madry et al., 2018] *Bold accuracies denote the unreliable robustness estimation cases.

Table 3: Unreliable Cases of Adversarial Testing Accuracy

the model true robustness property with different defenses. Under all the above cases, the robustness metric gives reliable robustness estimation, remaining un-affected by defense methods and the unreliable PGD adversarial testing accuracy.

5.4 Reparemeterization Invariance Evaluation

The reliability of our proposed metric is also reflected in its invariance from the model parameter scaling. Previous work [Keskar et al., 2017] tried to define a metric, named ϵsharpness, to evaluate the loss surface's geometry properties. We adopt its original definition and apply it into our input space to evaluate sharpness of input space loss surface, which can empirically reflect the adversarial generalization as aforementioned.

The experiment results are shown in Table. 4, where ϵ denotes the ϵ-sharpness, ψs denotes our robustness metric based on softmax layer without normalization, and ψn denotes our robustness metric with normalization. For the test cases, Org. indicates the tests with the original model without reparemeterization, *100 and /100 denote the model's logit layer weights and biases are scaled accordingly. Please note that, such scaling won't introduce accuracy and robustness change in practice [Dinh et al., 2017].

The experiments show that, both ϵ-sharpness and unnormalized ψs give very distinct robustness estimations influenced by the reparameterization. By contrast, the normalization method successfully alleviates the scaling influence and enables our metric ψn to keep a stable estimation under model reparameterization. Therefore, our metric could thus be used to more precisely capture one model's robustness degree without being affected by model reparameterization.

5.5 Efficiency of the Robustness Metric

Here we show the efficiency of our metric compared to adversarial testing methods. Since we are evaluating the model properties, theoretically it should be invariant to how many input we choose. Here we show that as the test-batch-size increases, the calculated robustness metric gradually converge to a stable robustness estimation which is close to the whole test set average robustness. Figure 6 shows the relation with the batch size and the robustness deviation between batches with same batch-size. We can see that on both datasets, as the batch size increases, the robustness measurement become more accurate since they have much smaller deviations. With the batch-size equals to 1000 (or less), we could get the model's robustness estimation with less than 10% deviation on MNIST and 5% on CIFAR10, which demonstrate higher efficiency than accuracy testing running on the whole test set.

5.6 Robustness Estimation Grading

Based on our experiments, we could derive a rough relationship between different robustness evaluation score and the adversarial accuracy. For example, on MNIST dataset within common threat model (ℓ∞ < 0.3), we can define the model robustness by three levels: Vulnerable (acc < 0.3), Fairly Robust (0.3 ≤ acc < 0.6) and Robust (0.6 ≤ acc ≤ 1.0). In such a case, the corresponding robust metric range will be (0, 100), (100, 270), (270, ∞), which could be used to quickly grade a neural network's robustness. The robustness grading for CIFAR and ImageNet cannot be well developed yet due to the limited robustness currently (40% and 15%).

6 Conclusion

In this work, through visualizing and interpreting neural networks' decision surface in input space, we show that adversarial examples are essentially caused by neural networks' neighborhood under-fitting issue. Oppositely, robust models manage to smoothen their neighborhood and relieve such under-fitting effect. Guided by such observation, we propose a model intrinsic robustness evaluation metric based on the model predictions' maximum KL-divergence in a given neighborhood constrain. Combined with our new-designed normalization layer, the robustness metric shows multiple advantages than previous methods, including: great generality across dataset/models/attacks/defenses, invariance under reparameterization, and excellent computing efficiency.

Batch Size (100 to1000)