Sensitivity Analysis of Multilayer Perceptron

1 Introduction

The sensitivity analysis of neural network has been investigated for over 30 years. In 1962, Hoff used ndimensional geometry approach to analyze the sensitivity of Adaline [Hoff, 1962]. Glanz [1965] simplified Hoff's results. After two decades, Winter [1989] derived an analytical expression for the probability of error in Madaline caused by weight perturbation. Stevenson continued Winter's work, and established the sensitivity of Madaline to weight error in 1990 [Stevension, 1990a; Stevension et al., 1990b]. Cheng and Yeung [1999] use this geometrical arguments to analyze the sensitivity of Neocognitrons. However, the their results are only fit for neurons with threshold activation function and binary inputs, and they are not applicable to other continuous activation functions. Yeung and Wang [1999] try to apply the method proposed in [Cheng and Yeung, 1999] to the sensitivity analysis of the MLP. Earlier in 1992, Choi and Choi [1992] present a thorough idea of sensitivity analysis of the MLP with differentiable activation functions. Although their analysis is systematic, the result is only applicable when the network has been trained and the inputs are known, so it cannot be used for network design and construction.

* This research work is supported by a Hong Kong Polytechnic University research grant number G-S546 and a Hong Kong CERG number 5114/97E

Piché [1992;1995] uses statistical approach to relate the output error and the change of weights in Madaline with several activation functions such as linear, sigmoid and threshold. But Piché's stochastic model is a simplification of the true MLP, and his result holds true only when the input and weight perturbations are small enough.

This paper proposes an improved stochastic model which is better suited for the true MLP. We also present a function approximation method to derive an universal analytical expression of sensitivity for MLP with sigmoidal activation function. We assume no restriction on the amplitude of the input and weight perturbations. Using this function approximation method, one can use three parameters to characterize the activation function. So not only the effects of input and weight perturbations, the number of neuron and the number of layer, but also that of the activation function on the sensitivity of MLP can be analyzed by the universal analytical expression.

2 The Improved Stochastic Model of MLP

According to Piché's stochastic model, all neurons have the same activation function. All network inputs, weights, input perturbations and weight perturbations are respectively independent identically distributed random (iidr) variables with zero mean; inputs and input errors, inputs and weight errors, weights and inputs, weights and weight errors are statistically independent from each other. This model has the following weaknesses:

1. The bias input is omitted. This is a basic component of MLP;
2. The expectations of all network inputs are zeros. Many applications don't satisfy this condition;
3. All network inputs have the same variance. It is also difficult for many applications to satisfy this requirement;
4. The correlations between the inputs and their corresponding perturbations are not considered. Even if the inputs and their associative perturbations in the first layer are independent, there still exist correlations between the inputs and their corresponding perturbations in the successive layers of MLP.

Our improved stochastic model of MLP overcomes the above shortcomings in the following ways:

1. All network inputs are independent random variables with their own expectations and variances. The expectation of the bias input is 1, and its variance is 0;
2. All input perturbations are independent random variables with zero mean and their own variances. The variance of the bias input perturbation is 0. The correlations between network inputs and their corresponding perturbations are expressed by their correlation coefficients;
3. All weights and their perturbations are independent random variables with zero mean and their own variances. The correlations between weights and weight perturbations are also expressed by their correlation coefficients;
4. Except the correlations between inputs and input perturbations, weights and weight perturbations, all variables are independent from each other.

Our model is a better description of the true MLP.

3 Definitions

First, some notations are introduced. The layers are indexed from 0 to L. The 0 layer is the input layer, and the Lth layer is the output layer. The number of nodes in layer l is denoted by l N . The function ) (⋅ g denotes the activation function. Neuron is the basic element of MLP, so one can first investigate the sensitivity of a single neuron, and then the whole MLP.

Suppose l X , l X ∆ are the input and input perturbation vectors of neuron i in layer l, l i W , l i W ∆ are its weight and weight perturbation vectors. D(.) denotes the variance.

Definition 1: The absolute sensitivity of the neuron to input and weight perturbations is given by

Definition 2: The relative sensitivity of the neuron to input and weight perturbations is

Since the outputs of MLP are the outputs of the neurons in the output layer, we can use the output neurons to represent the outputs of the MLP.

Definition 3: The absolute sensitivity of MLP to input and weight perturbations is

Definition 4: The relative sensitivity of MLP to input and weight perturbations is

4 Function Approximation Approach

If the sensitivity is directly calculated by using the above definitions, it will be very difficult to obtain an analytical expression for the sensitivity unless we restrict the amplitudes of the input and weight perturbations. But this would severely limit the range of application of our sensitivity analysis. So a function approximation approach is proposed to overcome this problem. This is based on the idea that a given class of activation functions can be characterized by a general function expression using a set of common characteristics, i.e., parameters of this expression, which should satisfy the followings:

1. They can be used to approximate the activation function concerned, and
2. They should have the form convenient for sensitivity
3. computation.

The threshold and sigmoidal activation functions are commonly used for the MLPs. It can be characterized by three factors: its maximum value, minimum value and obliquity, which are denoted by B, C and A respectively, Different values of B, C and A represent different activation functions. The main idea of the function approximation approach is to use another function ) ( , , x g C B A to approximate

) (x g . The function ) ( , , x g C B A has the following form

where

The objective of function approximation is to determine the three factors A, B and C of ) (x g , so that the distance between the two functions ) (x g and ) ( , , x g C B A is minimized.

The factors B and C are determined by

The factor A can be found by solving the following problem

Because ) (x g and ) ( , , x g C B A are antisymmetric about a certain point on the y-axis, problem (5) is equivalent to

The commonly used sigmoidal activation function has the form x x Ce B x g λ λ - - + ) ( , we have 2 π = A .

5 Sensitivity of a Single Neuron

According to the definitions of absolute and relative sensitivity, we need first derive the variances of the output and the output error of a neuron. The output of the unit j in layer l is Its output error is

) ( )) ( ) (( l l i l l T l i l i l i X W g X X W W g y ⋅ - ∆ + ⋅ ∆ + = ∆ . (8) The covariances between inputs and input perturbations, weights and weight perturbations are l x l x l x x l x x i i i i i i ∆ ∆ ∆ = σ σ ρ σ and l w l w l w w l w w ij ij ij ij ij ij ∆ ∆ ∆ = σ σ ρ σ , where ρ denotes the correlation coefficient.

Let 2 σ and µ represent variance and expectation. We have

According to the Lindeberg Central Limit Theorem, when 1 - l N is large enough, u and v' are normally distributed with mean 0 and variance 1, i.e., u ~ N (0,1) and v ~ N (0,1). The correlation coefficient between u and v is

Let

we get ) 1 , 0 ( ~ N V , and 0 , = V U ρ .

We introduce two notations

The distribution function of u, and the joint distribution function of u and v may be expressed as

Then

Now, the expectations of output and output error are

The variances of output and output error are

Using our function approximation approach and the following polar coordinate transformation

Formulae (25), (26), (27) and (28) become

According to Definitions 1 and 2,

where, let T=

For each neuron in a given layer, its inputs and the associated errors are just the outputs and their associated errors of the former layer. So l j l j x y = -1 , l j l j x y ∆ = ∆ -1 . Now the covariance between 1 - l j y and 1 - ∆ l j y can be calculated by

where

All formulae are derived exactly except the approximation of the activation function. So there is no need for us to restrict the input and weight perturbations to be very small. The network sensitivity expressions are expressed in terms of A, B and C, i.e., the activation function itself. This enables us to investigate the effect of a given activation function on the network sensitivity.

6 Sensitivity of MLP

Since the sensitivity of a MLP is defined to be the sensitivity vector of its output neurons, it will be necessary to calculate the sensitivity of neurons from the input layer to the output layer successively.

The algorithm to calculate the sensitivity of the network is given by

1. Based on the application requirement, determine the expectations and variances of network inputs 0 i x µ , 2 0 i x σ , input perturbation ratios 2 0 2 0 i i x x σ σ ∆ , the

correlation coefficients between inputs and input perturbations 0 i i x x ∆ ρ , the variances of weights 2 l wij σ ,

weight perturbation ratios 2 2 l w l w ij ij σ σ ∆ and the correlation

coefficients between weights and weight perturbations l w w ij ij ∆ ρ ;

1. Using function approximation approach to determine A, B and C of the activation function;
2. Let l=1, 2 2 x l x σ σ = , 2 2 x l x ∆ ∆ =σ σ ;
3. For all neurons in layer l, calculate 2 l yi σ and 2 l yi ∆ σ by applying formulae (32) and (33), and then l y y i i∆ σ , by Formula (39);

1. Calculate L i S and L i S l. Re by formulae (34) and (35), then the absolute and relative sensitivities of MLP is T L N L L T L N L L l l S l S l S l S l S S S S ) . Re , , . Re , . (Re . Re , ) , , , ( 2 1 2 1 ! ! = = , and the program is terminated.

7 Effects of Parameters on Sensitivity

Many factors affect the sensitivity of the MLP. These factors include the network inputs and input perturbations, weights and weight perturbations, and network architecture parameters such as the number of layer, the number of neurons in each layer and the activation function chosen. How these factors affect the sensitivity of MLP is an important criteria for neural network design in support of a particular application.

The experimental results are shown in figures 1 to 13. The main results are:

1. The input and weight perturbation ratios are the two most important factors affecting the sensitivity of the MLP. The absolute and relative sensitivities increase when these two ratios do.
2. The network sensitivities increase with the absolute value of the input expectation and the input variance. The normalization of network input can help us decrease the network sensitivity;
3. The increase of weight variance causes the network sensitivity to increase. So we should limit the amplitude of the weight in training process;
4. When the number of neurons increases, the absolute and relative sensitivities increase, but when it is significantly large, the two sensitivities will nearly remain constant. The neuron number of later layers has more influence upon the absolute sensitivity than that of the previous layer.
5. The absolute and relative sensitivities increase with the number of layer. When the number of layer is small, these two sensitivities increase rapidly. Otherwise they become flat. When the weight perturbation ratio increases, the absolute and relative sensitivities increase as well, i.e., if the sensitivity is supposed to be less than a certain value, the number of layer could be small when the weight perturbation ratio is large.
6. The larger the obliquity A of the activation function is, the more sensitive the MLP is. The sensitivities increase with B-C. So the threshold MLP is the most sensitive one, and The activation function x e - + 1 1 ��

more suitable than tanh(x).

Figure 4: The absolute and relative Sensitivities of MLP versus the variance of network input.

Figure 5: The absolute and relative Sensitivities of MLP versus the weight variance.

Figure 6: The absolute sensitivity of MLP versus the number of neurons in each layer.

B

Figure 10: The relative sensitivity of a neuron versus the max and min value of the activation function.

Figure 11: The absolute sensitivity of MLP versus the number of layer.

Figure 12: The relative sensitivity of MLP versus the number of layer.

8 The Design of MLP

In order to simplify the network design, we suppose that all weights have the same variances, and all weight perturbation ratios are identical. So all neurons in the same layer have the same sensitivity. We can use a output neuron's sensitivity to represent the network sensitivity vector.

For a given application, suppose parameters L, B and C are determined by the network designer, from the sample set

C

and the application environment, we can estimate the following parameters: 0 N , L N , 0 i x µ , 2 0 i x σ , 0 , i i x x ∆ ρ , 2 0 2 0 i i x x σ σ ∆ ,

w w ∆ , ρ , 2 2 w w σ σ ∆ . Our goal is to find the unknown parameters

1 N , …, 1 - L N , A and 2 w σ , such that * R R S S < , where * R S is a given, acceptable relative sensitivity.

Our solution is: first select a possible range for each of the unknown parameters; then using our sensitivity calculation algorithm to compute the mapping from } , , , , { 2 1 1 w L A N N σ - ! to } { R S ; finally, according to the mapping, determine the region } , , , , { 2 1 1 w L A N N σ - ! that satisfies * R R S S < .

We present the following example to show how to design a three-layer MLP.

Example: The input and output numbers are 10, and all inputs are independent random variables that are uniformly distributed over [-1,1]. Suppose the correlation coefficient between weights and weight perturbations are 0.01. Construct a three-layer MLP with input and weight perturbation ratios less than 1%, and the relative sensitivity of this MLP should be less than 10%.

For this example, the known parameters are L=2, 10 2 0 = = N N , 0 0 = i x µ , 12 / 4 2 0 = i x σ , 0 0 , = ∆ i i x x ρ ,

C=0, and the possible ranges for parameters A, 1 N and 2 w σ are 1.7725] [0, , 100] [1, and 25] [0, respectively. Using our sensitivity algorithm to calculate the mapping over the above ranges. Here, we plot two contour maps of R S . When the obliquity A is large and small, the contour lines of R S with respect to 1 N and w σ are shown in figures 13 and 14 respectively. The region below the contour line at 1 . 0 = R S is our design result.

Our sensitivity analysis can be used to determine the network structure numerically. Furthermore, it can estimate the permitted weight range for network training. We find that the larger 1 N is, the smaller the maximum permitted weight variance is. That is to say, if we select larger number of neurons in the hidden layer, we must restrict the weight variance to be less in the training of MLP; when the obliquity of activation function increases, the solution region reduces. This means that if we select activation function with smaller obliquity, we can permit larger number of neurons in the hidden layer and larger weight variance in the network training.

σw

9 Conclusion

In this paper an improved stochastic model of MLP is proposed to better describe the, and the function approximation method is used to obtain an universal expression of the network sensitivity for sigmoidal activation functions. We use the expression of sensitivity to investigate the effects of network parameters on the network sensitivity. It is also used to design the network architecture and provide useful guideline for the network training.