Optimal ANN-SNN Conversion for Fast and Accurate Inference in Deep Spiking Neural Networks

1 Introduction

As a representative of artificial intelligence methods, deep learning has begun to exceed or approach human performance in various tasks, including image classification, natural language processing, and electronic sports [He et al., 2016a; Brown et al., 2020; Berner et al., 2019]. But this success is at the cost of high energy consumption. Recently, neuromorphic hardware, including TrueNorth, SpiNNaker, Loihi, and so on [DeBole et al., 2019; Painkras et al., 2013; Davies et al., 2018], is attracting more and more researchers due to their high temporal resolution and low power budget. This kind of hardware runs Spiking Neural Networks (SNNs) instead of Artificial Neural Networks (ANNs). With unique memory and communication designs, an SNN implemented on SpikNNaker can achieve the power consumption of 0.3W for MNIST classification [Stromatias et al., 2015]. For object detection, the spiking version of a YOLO model is estimated to be at least 100 times more energy-efficient than that on GPU [Kim et al., 2020]. Such dedicated hardware and algorithms are highly appealing for mobile applications like autonomous vehicles and other power-limited scenes.

∗Corresponding author

Nevertheless, training high-performance SNNs is a nontrivial problem. The neurons in SNNs emit discrete spikes, which disables the direct backpropagation training. Up to now, the training algorithms for SNNs can be summarised into four methodologies: supervised backpropagation through time [Wu et al., 2019; Zenke and Vogels, 2021], unsupervised STDP learning [Kheradpisheh et al., 2018; Diehl and Cook, 2015], ANN-SNN conversion [Cao et al., 2015; Diehl et al., 2015; Rueckauer et al., 2017], and other mixture methods [Lee et al., 2018; Tavanaei and Maida, 2019; Rathi et al., 2020]. For deep SNN training, ANNSNN conversion requires less GPU computing than supervised training with surrogate gradients. Meanwhile, it has yielded the best performance in large-scale networks and datasets among methodologies. Therefore, ANN-SNN conversion has become the first choice for deep SNN training, which is also the focus of this paper.

As illustrated in Fig. 1, ANN-SNN conversion is to map the parameters of a pre-trained ANN to an SNN with low accuracy loss. Cao et al. [2015] started the study of ANNSNN conversion. They found the equivalence between the ReLU activation and the spiking neurons' firing rate, which is the foundation of later rate-based methods. Diehl et al. [2015] attributed the performance loss to inappropriate activation of neurons and proposed Weight Normalization methods (model-based and data-based) to scale the ANN weights. Rueckauer et al. [2017] gave a detailed theoretical explanation of ANN-SNN conversion and proposed a new reset-bysubtraction neuron to overcome accuracy degradation. They also extended the use of bias and Batch Normalization (BN) and proposed the Max Normalization algorithm (Max Norm for short), which uses maximum values of activation as scaling factors. Kim et al. [2020] suggested to use channel-wise normalization for convolutional nets. Different from conversion using ANN activation, Sengupta et al. [2019] proposed SpikeNorm, which makes use of spiking statistics to set the thresholds. To enable more accurate conversion, Rueckauer et al. [2017] further proposed Robust Normalization (Robust Norm for short), where the scaling factor changes from maximum activation value to 99.9% of activation. Yet this is not the first attempt to manually manipulate the threshold or factor. Cao et al. [2015] set the firing thresholds based on spike density. These practices allow the firing rates of some neurons to be constant 1. Here we refer to this phenomenon as spike saturation. However, both the maximum and 99.9% are rigid, which inspires us to explore a trainable way to achieve low conversion loss.

The conversion methods mentioned above all incur long simulation time when applied to deeper networks and more complicated datasets. That is, converted SNNs need a longer time to rival the original ANNs in precision. This restricts the practical promotion, such as real-time tracking and detection. Robust Normalization somewhat mitigates this problem by increasing the firing rates. Spike saturation actually causes the subsequent layers to infer faster. Based on this observation, Han et al. [2020] started to improve the inference latency by scaling the SpikeNorm thresholds. Han et al. then gave a theoretical analysis of the scale (setting the firing threshold as the expectation of weights × spikes), but they used the manually set value eventually. Nevertheless, this inspires us that inference latency and parameters can establish associations on the model. A hybrid training scheme also helps. Rathi et al. [2020] realized fewer inference time-steps by conversion-based initialization and spiketiming-dependent backpropagation. Other methods concern coding schemes to achieve fast inference (i.e. shorter inference time), including Temporal-Switch Coding [Han and Roy, 2020], FS-conversion coding [Stöckl and Maass, 2021]. However, simply bypassing rate coding is not that rational. Though rate coding is not the perfect coding scheme, it is partially in line with observations in the visual cortex [Rullen and Thorpe, 2001]. Therefore, fast inference for rate-encoding deep SNNs is an important research direction. But it still lacks instructive principles and theories.

In this paper, we propose an ANN-SNN conversion method that enables high accuracy and low latency. The main contributions of this paper are summarized as follows:

• We theoretically analyze ANN-SNN conversion and derive the sufficient conditions of the optimal conversion. Based on this, we propose Rate Norm Layer to replace the ReLU activation function in source ANN, enabling direct conversion from a trained ANN to an SNN. This will reduce the potential loss of information caused by normalization.
• We propose an optimal fit curve to quantify the fit between the activation value of source ANN and the actual firing rate of target SNN, and derive one upper bound of this convergent curve. We show that based on the Squeeze Theorem, the inference time can be reduced by optimizing the coefficient in the upper bound. These results can not only systematically explain previous findings that reasonable scaling of the threshold can speed up inference, but also give a proper theoretical basis for fast inference research.
• We demonstrate the utility of the proposed method with near loss-less conversion in deep network architectures on the MNIST, CIFAR-10, CIFAR-100 datasets. Moreover, it achieves 8.6× faster reasoning under 0.265× energy consumption of the typical method.

2 Methods

In this section, a theory for ANN-SNN conversion is first introduced. Based on this, Rate Norm Layer with trainable threshold is thus proposed. Then, we analyse the reason for slow inference and suggest optimization for fit of firing rates. Finally, we present a stage-wise learning strategy for accurate and fast SNN.

2.1 Theory for Conversion fromANN to SNN

The fundamental principle of ANN-SNN conversion is to match analog neurons' activation with spiking neurons' firing rate. One common way is to convert ReLU nonlinearity activation to the Integrate-and-Fire (I&F) neuron. To be specific, for analog neurons in layer l (l = 1, 2, ..., L), the ReLU activation can be described by:

where vector al is the output of all ReLU-based artificial neurons in layer l, Wl-1 and bl-1 is the weight and the bias term for the neurons in layer l - 1.

As for the I&F neuron, the membrane potential vi l (t) for the i-th neuron in layer l is formulated by:

where bil-1 denotes the bias current to the i-th neuron, W ij l-1 denotes the synaptic weight between the j-th presynaptic neuron in layer l-1 and the i-th neuron in layer l. δ(·) is the delta function. Tj denotes the set of spike time of the j-th presynaptic neuron, i.e., Tj = {t(1) j , t(2) j , . . . , t(K) j }. When the membrane potential vi l (t) exceeds the firing threshold vth,l in layer l, a spike is generated and the membrane potential vi l (t) is reset to the rest value vrest < vth,l.

To match analog neurons' activation with spiking neurons' firing rate, we discretize and vectorize Eq. 2 into time-steps and obtain the spiking neuron model for layer l.

where ml(t) and vl(t) represent the membrane potential of all I&F neurons in layer l after neuronal dynamics and after the trigger of a spike at time t, U(·) is the Heaviside Step Function, sl(t) denotes the vector of binary spikes, the element in which equals 1 if there is a spike and 0 otherwise. bl-1 is the vector of bil-1 , and Wl-1 is the weight matrix. Note that here we use the "soft reset" [Han et al., 2020] instead of the "hard reset". At the moment of a spike, the membrane potential vl(t) is reduced by an amount equal to the firing threshold vth,l, instead of going back to the reset value.

Based on these definitions, we can derive the relationship between the firing rate rl(t) of spiking neurons in layer l and rl-1(t) of neurons in layer l - 1, which is depicted in Lemma 1. The proof can be found in the Appendix.

Lemma 1. For a spiking neural network consisting of the reset-by-subtraction neurons mentioned in Eq. 3, assume that Wl-1 and bl-1 are the parameters for layer l-1. Then when t → ∞, the relation of the firing rate rl(t) and rl-1(t) is given by:

where clip(x, 0, 1) = x when x ∈ [0, 1], clip(x, 0, 1) = 1 when x > 1, and clip(x, 0, 1) = 0 when x < 0.

With Lemma 1 and Eq. 1, We can derive the theorem for conversion from ANN to SNN:

Theorem 1. For an L-layered ANN with the ReLU activation and an L-layered SNN with the reset-by-subtraction neurons, assume that W ANN l-1 , bANN l-1 are the parameters for layer l-1 of the ANN, and W SNN l-1 , bSNN l-1 are the parameters for layer l-1 of the SNN. maxl is the maximum activation of layer l in ANN, and vth,l is the firing threshold of layer l in SNN. The ANN can be converted to the SNN when t → ∞ (Eq. 1 equals Eq. 4) if for l = 1, 2, ..., L, the following equations hold:

The proof of Theorem 1 is presented in the Appendix. Eq. 5 implies that scaling operations are necessary to convert ANN to SNN, either scaling weights (i.e. weight normalization) or setting thresholds (i.e. threshold balancing). In this sense, weight normalization is equivalent to threshold balancing.

2.2 Rate NormLayer

The choices of scaling factors are often empirical, and posttraining [Rueckauer et al., 2016; Han et al., 2020]. To overcome this, we propose Rate Norm Layer (RNL) to replace the ReLU activation in ANN. The idea is to use a clip function with a trainable upper bound to output the simulated firing rate, which is the limitation of actual firing rate in SNN when inference time T → ∞. Here we denote the simulated firing rate as ˆ rl , Rate Norm Layer can be formally expressed as follows:

where pl is a trainable scalar (pl ∈ [0, 1]), and θl is the threshold of the l-th layer. With Theorem 1 satisfied (vth,l = θl) and pl = 1, one can find that Eq. 6 is equivalent to Eq. 4. In this case, RNL will degenerate to the Max Norm algorithm, which scales the weight Wl-1 by maxl-1 maxl and the bias bl-1 by 1 maxl . A diagram comparison of different scaling schemes can be seen in Fig. 2. For mini-batch training, different batches have different maximum outputs. To reduce the perturbation caused by data sampling, running_max(Wl-1ˆrl-1 + bl-1) is used instead of max(Wl-1 ˆ rl-1 +bl-1). The design of the Rate Norm Layer mainly considers the following three factors:

• Compared with directly cutting the simulated firing rate to 1, the backpropagation becomes more effective in RNL training. max(Wl-1ˆ rl-1 + bl-1) and running_max(Wl-1ˆrl-1 + bl-1) can enable the gradient to flow out smoothly. Their participation in threshold calculating is similar to Batch Norm (BN) in mini-batch training. However, RNL cannot replace BN, because BN rescales the data to a normal distribution, which is related to the characteristics of ANN.
• The threshold θl enables better generalization. Existing solutions mainly focus on using a subset of the training data for offline normalization. This will potentially influence the generalization of the scaled SNN for data out of subset. In contrast, θl uses all data in training, which can be used directly in SNN inference.
• The threshold θl is determined in training. For faster inference, Robust Norm requires empirical percentile which RNL doesn't need, as pl is trainable. Certainly, just using ANN loss will not guide the model to reduce

T

Figure 3: Firing rate curves of output neurons in a toy SNN with different rate coding schemes. The membrane potential of spiking neurons has been randomly initialized from independent samplings.

the inference delay. This also requires additional loss design, which will be shown later.

2.3 Optimization for Fast Inference

Rate-based SNN models take rate coding as input, and the time average of the output spikes as output. In the conversion methods, the coding scheme mainly consists of two ways. One is Poisson coding, of which spikes obey the Poisson process. The other is constant coding, to cooperate with the reset-by-subtraction neuron. Constant coding is not a new thing, it can be regarded as an integrating ADC in the signal processing [Eng Jr and Matson, 1994]. The two primary rate coding forms take time for the firing rate to approach its expectations. For the constant coding, its accumulated current needs to be rounded down when converted to spike counts. So the firing rate will be jagged and approach the analog value (Fig. 3). Using both codings will bring about unpredictable rate output in the first few time-steps. Following the suggestions of previous literature, constant coding is chosen as the primary scheme in this paper.

The time when the output firing rate of an SNN matches the analog output of an ANN is referred to as "inference time", "inference latency" or "inference delay" [Neil et al., 2016]. Fig. 3 implies that for both rate coding schemes, there will be an output delay. In deep neural networks, the stacking of layer-by-layer delays will bring greater inference delay. For example, ResNet-44 requires 350 time-steps to achieve the best accuracy [Hu et al., 2018]. This problem is not limited to ANN-SNN Conversion. The BPTT trained SNN model also has similar issues.

Now that the reason for slow inference is attributed to the deviation of the encoder accumulation. To further analyze the characteristics, we propose to use K(ˆ r, r(t)) (K curve) to quantify the relationship between the simulated firing rate ˆ r of ANN and the real firing rate r(t) of SNN after conversion.

Note that the design of K resembles the chi-square test in hypothesis testing. ˆ r and r(t) denote the firing rates of all neurons in a certain layer. ‖ · ‖2 indicates L2 norm. The denominator ‖ˆ r‖22 makes K(ˆ r, r(t)) have scale invariance. Therefore, we can compare the fitting of the firing rate between different layers. Ideally, given enough "inference time", K(ˆ r, r(t)) will converge to 0. We believe K curve is an adequate metric as the neuron population encoding information is considered rather than any single neuron.

Specifically, Fig. 4 gives an example to illustrate how the K curve fits between different layers of a VGG16 and a converted SNN. An image is used for reasoning and calculating K curves. As the layer deepens, the convergence speed of the K curve becomes slower. By accelerating the convergence of the K curve, the inference can speed up. Here we derive one of the upper bound for K(ˆ r, r(t)).

Theorem 2. For layer l in an ANN and the converted SNN with constant coding, given the simulated firing rate ˆ rl and the real firing rate rl, we have:

where Kl denotes the abbreviation of K(ˆ rl, rl(t)) in layer l, Ωl = ‖ˆ rl‖1 ‖ˆ rl‖22 . ‖ · ‖p denotes Lp norm.

The detailed proof of Theorem 2 is described in the Appendix. Ωl is named as Rate Inference Loss (RIL) of the l-th layer. Eq. 8 indicates that Kl is less than an inverse proportional curve to t.

For the convergence of the last layer (the L-th layer), if ΩL reduces, KL will converge faster to 0 due to the Squeeze Theorem. That is, the real firing rate of the SNN approaches the simulated value of the ANN more faster, leading to faster and more stable outputs of SNN. However, considering that the network actually has inference delays layer by layer, a better solution is to reduce the average value of Ωl(l = 1, 2, . . . , L). Thus, the overall training loss is composed of the loss related to the task and RIL multiplied by hyperparameter λ.

where (x, y) is data tuple for training and f(·) is the network with L Rate Norm Layers. Based on Eq. 6, we are able to calculate the partial derivative of ΩL w.r.t. pL, and obtain:

where ˆ rL,i denotes the i-th element of ˆ rL.

Eq. 10 implies that the partial derivative of ΩL w.r.t. pL is positive. Simply minimizing ΩL will reduce the neurons' pl. The upper limit of pl is unnecessary in this sense. Nevertheless, this will lead more neurons to saturation state and lose model accuracy. Thus, we jointly optimize the two losses and tune the hyper-parameter λ to reach the optimal trade-off between model accuracy and conversion loss.

So far, the current theories and analyses can also systematically explain the findings of Han et al. That is, reasonable scaling of the threshold can speed up the inference. If the threshold is set smaller, the Rate Inference Loss will decrease. Since the curve value and time are inversely proportional, it is equivalent to accelerating the decrease of the curve. On the contrary, if the threshold is too large, many neurons in the SNN will have low firing rates to produce accurate output. Of course, the threshold should not be too small to prevent more neurons from saturating and losing information. Fig. 4(b) shows the K curve after adjusting the threshold using the algorithm of Han et al.

2.4 Training for Accurate and Fast SNN

In Section 2.2, the pl of the Rate Norm Layer is restricted to [0, 1]. This means if the Rate Norm Layer is directly trained, the simulated firing rate after clipping and scaling will inevitably bear information loss. The information loss here indicates that the output distribution due to cropping is different from the original distribution. This will make it difficult to take advantage of ANN's performance. So the training strategy needs to be carefully designed. When pl = 1 is fixed, zl is clipped with running_max(zl), which has less information loss. After the network synaptic parameters are fixed, the neuron threshold starts training. The training goal at this stage is to reduce information loss and to reduce Rate Inference Loss. As the output f(x) of an ANN is ˆ rL, the goal is to optimize (according to Eq. 9):

we decompose this goal into two stage goals (shown in Fig. 5):

Stage 1 is accuracy training, when network outputs ˆ r∗L. The target is:

Stage 2 is for fast inference. Assume the output of this stage is ˆ r′L. Then the target is:

The cos distance is used to maintain neuron information. The detailed training and converting algorithm is described in Algorithm 1 in the Appendix.

3 Experiments

3.1 Experiment Implementation

We validate our methods on the image recognition benchmarks, namely the MNIST 1 , CIFAR-10, CIFAR-100 2 datasets. For MNIST, we consider a 7-layered CNN and AlexNet. For CIFAR-10, we use VGG-16 and PreActResNet18 network structures. It is worth noting that we did not use the common ResNet as we think PreActResNet will help the training of Rate Norm Layers [He et al., 2016b]. For CIFAR100, VGG-16, PreActResNet-18 and PreActResNet-34 are used. We present the simulation results and analysis in the following subsections.

3.2 Accuracy Performance

We first evaluate the effectiveness of the proposed Rate Norm Layer. A network with Rate Norm Layers is trainable with backpropagation. When testing SNN performance, all Rate Norm Layers are converted to I&F neuron models with θl as the threshold. Table 1 shows the best accuracy of the converted SNNs compared with typical works. The converted SNNs achieve the state-of-the-art performance on MNIST and CIFAR-100 datasets, and reach a similar performance on CIFAR-10. For VGG-16 trained by CIFAR-100, the proposed method reaches top-1 accuracy 75.02%, whereas the state-of-the-art ANN-SNN algorithm reaches 70.93%. Conversion loss is considered here to evaluate the quality of the ANN-SNN conversion. As illustrated in Table 1, ANN-SNN conversion with Rate Norm Layers has low conversion loss, or even negative conversion loss, indicating that the converted SNN may outperform the original ANN. In contrast, the loss is usually positive for other methods, meaning that the performance of the converted SNN is not as good as ANN.

3.3 Fast Inference Performance

We test whether the proposed Rate Inference Loss can speed up inference, that is, speed up the convergence of the K curve. Fig. 6(a) and (b) show how the K curve and accuracy change over latency, where the dotted line in Fig. 6(a) represents the Max Norm method. As can be seen from Fig. 6(a), the K curves of the proposed method converge to 0 quickly, which are much faster than those of the Max Norm method. Thus the proposed method can implement fast inference. The inference performance can be observed in Fig. 6(b). The proposed method reaches an accuracy of 85.40% using 32 timesteps, whereas the methods of Max Norm, Robust Norm, and RMP-SNN reach 10.00%, 43.03% and 63.30% at the end of 32 time-steps. Moreover, the proposed method achieves an accuracy above 90% using only 52 time-steps, which is 8.6 times faster than Max Norm that uses 446 time-steps. Detailed accuracy comparison on time T is shown in Table 2.

The threshold and Ω of VGG-16 are visualized in Fig. 6(c). For the Max Norm method, as all the I&F neurons use the same threshold 1, we regard the maximum value of ReLU outputs as the equivalent threshold. It can be found that the distribution gap of the threshold is relatively large. But when paying attention to the Ω distribution, the Ω after the threshold scaling (orange column) is usually smaller than that of Max Norm (blue column). In contrast, training with Rate Inference Loss will keep Ω at a relatively low and average level, and thus benefit inference.

3.4 Energy Estimation of Neuromorphic Hardware

SNNs have a considerable potential on neuromorphic chips. One of the benefits is to reduce the energy budget. To study the energy efficiency of fast reasoning, we use the energy model proposed by Cao et al. [Cao et al., 2015] to model the energy consumption on neuromorphic chips. Assume that a spike activity would bring about the energy consumption of α Joules and 1 time-step takes 1ms. Then the power model is defined as:

1 http://yann.lecun.com/exdb/mnist/

2 https://www.cs.toronto.edu/ kriz/cifar.html

The previous energy analysis mainly focused on total energy consumption during inference. However, in real application scenarios, the total energy consumed before the model reaches reliable accuracy is more important. In this regard, we evaluate the performance of the proposed method and the Max Norm method on energy. Fig. 7(a) is the power histogram over time. Due to the lower threshold, the power of our model is relatively high. The integral of the deep color area represents the energy consumed to achieve 90% accuracy. The energy consumption of the proposed model is only 0.265 times of the Max Norm method when it reaches 90% accuracy. This means that the 8.6 × reasoning speedup will not bring much more energy consumption. Fig. 7(b) shows

(a)

(b)

Figure 7: Energy analysis of VGG-16 on neuromorphic chips. (a) is the power histogram. The integral of the deep color area is the energy for accuracy to reach 90%. (b) is the logarithmic ratio of energy consumption and inference speedup. 'p' is for the proposed method, and 'b' is for the baseline Max Norm method.

the logarithmic ratio of energy and inference speedup. Our model exhibits the properties of "fast reasoning" and "energy efficiency".

4 Conclusions

This paper proposes a method to convert conventional ANNs to SNNs. The Rate Norm Layer is introduced to replace ReLU for optimal conversion. Besides, we quantify the fit between the ANN activation and the firing rate of the converted SNN by an optimal fit curve. The inference time can be reduced by optimizing the coefficient of the upper bound of the fit curve, namely Rate Inference Loss. Thus, a twostaged learning scheme is proposed to obtain fast and accurate deep SNNs. Experimental results demonstrate that our methods achieve low accuracy loss and fast reasoning with deep structures such as VGG and PreActResNet.

Acknowledgements

This work was supported by the National Natural Science Foundation of China (62027804, 61825101, 62088102 and 61961130392).

Appendix A: Proofs

Here we provide the theoretic proofs of theorems and lemmas in this paper.

Lemma 1. For a spiking neural network consisting of the reset-by-subtraction neurons mentioned in Eq. 3, assume that Wl-1 and bl-1 are the parameters for layer l-1. Then when t → ∞, the relation of the firing rate rl(t) and rl-1(t) is given by:

where clip(x, 0, 1) = x when x ∈ [0, 1], clip(x, 0, 1) = 1 when x > 1, and clip(x, 0, 1) = 0 when x < 0.

Proof. For the reset-by-subtraction spiking neurons formulated by Eq. 3 in the main text, we can stack the equations and get the discrete function between spikes of layer l and layer l - 1:

By summing the left and right expressions over time and dividing tvth,l on the both sides, the equation can be reformulated as:

where rl(t) denotes the firing rates of all neurons in layer l. From the membrane potential updating function (Eq. 3),

vl(t) is in the range of [0, vth,l], thus we have:

As the value of sl(t) can only be 0 or 1, the firing rate rl(t) is strictly restricted in [0, 1]. When t →∞, it's straightforward to conclude that

Theorem 1. For an L-layered ANN with the ReLU activation and an L-layered SNN with the reset-by-subtraction neurons, assume that W ANN l-1 , bANN l-1 are the parameters for layer l-1 of the ANN, and W SNN l-1 , bSNN l-1 are the parameters for layer l-1 of the SNN. maxl is the maximum activation of layer l in ANN, and vth,l is the firing threshold of layer l in SNN when t → ∞. The ANN can be converted to the SNN (Eq. 1 equals Eq. 4) if for l = 1, 2, ..., L, the following equations hold:

Proof. The conversion of ANN to SNN is built on the basis of the equivalent of SNN firing rate and ANN activation. Considering spiking neurons of layer l and l - 1 in an SNN, the relationship between the firing rate rl(t) and rl-1(t) is:

Note that the firing rate ri(i = 1, 2, . . . ,L) in Eq. 7 is restricted in [0, 1]. But the ANN activation (ReLU) only satisfy ai ≥ 0 (Eq. 1 in the main text). In fact, for countable limited dataset, the activation generated by the network is also upper bounded. Assume that the upper bound for the output of all ReLU-based artificial neurons in layer i is maxi, we have:

Let zi = ai maxi , then 0 ≤ zi ≤ 1 (i = 1, 2, . . . , L). According to Eq. 1, the activation of layer l and l - 1 satisfy:

As the al clipped by maxl equals the original al. Then,

By dividing maxl on both sides of Eq. 10 and substituting ai by zi maxi, we have:

Comparing Eq. 7 and Eq. 11, We can conclude that Eq. 7 equals Eq. 11 if for l = 1, 2, ...,L, the following equations hold:

Theorem 2. For layer l in an ANN and the converted SNN with constant coding, given the simulated firing rate ˆ rl and the real firing rate rl, we have:

where Kl denotes the abbreviation of K(ˆ rl, rl(t)) in layer l, Ωl = ‖ˆ rl‖1 ‖ˆ rl‖22 . ‖ · ‖p denotes Lp norm.

Proof. Consider that firing is a cumulative and rounded firing process. When ˆ rl is given, rl(t) is approximate as ⌊ˆ rl t⌋ t , also ˆ rl,i - 1t < ⌊ˆ rl,it⌋ t ≤ ˆ rl,i, where ˆ rl,i denotes the i-th element of the vector ˆ rl. For the l-th layer, we have:

The last equality holds as simulated firing rate ˆ rl ≥ 0. Thus the sum of all items in ˆ rl equals its L1 norm. Now we conclude that:

Appendix B: Supplementary of Methods and Experiments

The training and converting algorithm is described in Algorithm 1. In Stage 2, since each trainable pi will scale the output, to reduce the instability of threshold training on the deep model, all layers share the same pi when training. Besides, to limit pi ∈ [0, 1] of Rate Norm Layer, use sigmoid(p′i ) in place of pi.

The experiments are conducted on the PyTorch platform. The GPU used in training is NVIDIA GeForce RTX 2080 Ti.

Algorithm 1 Mini-batch training and converting a spiking neural network from a source ANN with Rate Norm Layers. 'r_max' is short for 'running_max'.

Require:A spiking neural network fSNN with thresholds {vth,k|k=1, 2, · · · ,L}; A network with L Rate Norm Layers fANN; Dataset D for training; Number of epochs for Stage 1 epoch1; Number of epochs for Stage 2 epoch2

Ensure: pk=1.0, r_maxk=1.0 (k = 1, 2, . . . , L); momentum parameter m = 0.1, λ = 0.5

1: {Stage 1: Accuracy Training for fANN} 2: for e = 1 to epoch1 do 3: for length of Dataset D do 4: Sample minibatch (x, y) from D 5: ˆ r∗ 0 = x 6: for k = 1 to L do 7: r_maxk = (1-m) r_maxk +mmax(ˆ r∗ k-1 ) 8: θk = pk r_maxk 9: ˆ r∗ k = clip[(Wk-1 ˆ r∗ k + bk-1)/θk,0,1] 10: end for 11: Loss = L (ˆ r∗L; y) 12: Backward propagation through network 13: Update Wk and bk(k = 1, 2, . . . ,L-1) 14: end for 15: end for 16: {Stage 2: Fast Inference Training for fANN} 17: for e = 1 to epoch2 do 18: for length of Dataset D do 19: Sample minibatch (x, y) from D 20: ˆ r∗ 0 = ˆ r′0 = x 21: for k = 1 to L do 22: r_maxk = (1-m) r_maxk +mmax(ˆ r∗ k-1 ) 23: θk = pk r_maxk 24: ˆ r∗ k = clip[(Wk-1 ˆ r∗ k + bk-1)/θk,0,1] 25: end for 26: for k = 1 to L do 27: r_maxk = (1-m) r_maxk +mmax(ˆ r′k-1 ) 28: θk = pk r_maxk 29: ˆ r′k = clip[(Wk-1 ˆ r′k + bk-1)/θk,0,1] 30: end for 31: Loss = T (ˆ r∗L, ˆ r′L) (Eq.13) 32: Backward propagation through network 33: Update pk(k = 1, 2, . . . ,L) 34: end for 35: end for 36: {Converting SNN fSNN from pre-trained fANN} 37: for k = 1 to L do 38: fSNN.Wk = fANN.Wk 39: fSNN.bk = fANN.bk 40: fSNN.vth,k = fANN.θk 41: end for 42: return fSNN