Improving Classification Accuracy of Feedforward Neural Networks for Spiking Neuromorphic Chips

1 Introduction

Neural networks achieve human level performance in many image analytics tasks but neural networks are mostly deployed to GPU platforms that consume a considerable amount of power. New hardware platforms using lower numerical precision achieve drastic reductions in power consumption. Brain-inspired spiking neuromorphic chips such as the IBM TrueNorth Neurosynaptic System [Merolla et al., Benjamin Scott Mashford IBM Research, VIC, Australia bmashfor@au1.ibm.com

2014] have achieved power consumption on the order of milliwatts while still offering high accuracy and real-time processing. Recent work [Esser et al., 2015; 2016] has used these ideas to implement constrained neural networks to enable high performance image analytics on TrueNorth [Merolla et al., 2014]. With recent advances in hardware platforms already available, there is an urge to provide software that fully exploits the potential of this new type of hardware.

Recent approaches in machine learning have explored training models constrained to binary weights [Courbariaux et al., 2015] or low precision arithmetic [Courbariaux et al., 2014; Cheng et al., 2015; Hwang and Sung, 2014] and spiking deep belief neural networks [Stromatias et al., 2015], achieving state of the art performance in image analytics tasks. Lower power consumption is achieved by using only additions (avoiding expensive multiplication modules), which can benefit from implementing these models in configurable architectures such as FPGA or brain-inspired systems.

Recent methods for training adaptations of fully connected style layers to the TrueNorth core-to-core connectivity [Esser et al., 2015] applied to EEG data have shown that results using neuromorphic platforms exhibit performance levels that are below current work using unconstrained hardware [Nurse et al., 2016]. In this work, we revise the training of deep neural networks for low energy neuromorphic architectures and demonstrate high performance levels comparable to nonconstrained neural networks on MNIST and EEG data sets.

The selection of datasets in this study is guided by emerging areas of application for neuromorphic computing platforms. It has been proposed that the development of highperformance classifiers that run within a constrained power consumption limit will enable a new generation of mobile sensing applications [Lane et al., 2015]. Early examples of such applications include healthcare-related wearables that involve machine vision and audio-signal processing [Poggi and Mattoccia, 2016], as well as real-time analysis of imagery for visual monitoring and surveillance [Antoniou and Angelov, 2016].

This paper is organized as follows. First feedforward neural networks and their training are introduced, followed by an overview of the TrueNorth chip architecture. Then a revised constrained model of the feedforward neural network training and deployment for TrueNorth are introduced. Finally results on EEG data and the MNIST handwritten data set are presented with discussion and future work.

2 Feedforward Neural Networks

In feedforward neural networks, inputs xi are integrated in units Ij using a set of weights wij and a bias term bj (which can be shared among units) as shown in equation 1. During training, the weights wij and bias bj are learnt.

Neurons output are a combination of the integration function Ij followed by a non-linearity, which can be a sigmoid function. The neurons can be grouped into layers and these layers can be stacked as shown in figure 1.

Prediction is done using the output layer outcome. In classification, typically each neuron is associated to one prediction category. Information is integrated in the output layer neurons, which are then be turned into probabilities using, for instance, softmax (eq. 2). pl is the prediction probability for category cl from all the possible K categories.

During training, weight and bias values are typically learnt using backpropagation [Duda et al., 2012], which minimizes a loss function that represents classification error of the network using gradient descent. An example of such a loss function is the logarithmic loss, which is used in this work (eq. 3), even though other functions could be explored.

The next sections describe the neuromorphic hardware and an explanation about how the neural networks can be constrained, learnt and deployed.

2.1 Deployment Hardware

IBM TrueNorth [Merolla et al., 2014], a low power and highly parallelized brain-inspired chip is used here as an example deployment system, though our method could be generalized to other neuromorphic hardware [Benjamin et al., 2014; Painkras et al., 2013; Pfeil et al., 2012]. In its current implementation it is composed of 4,096 neurosynaptic cores.

Figure 2 shows the layout of one of these cores. Each core has highly configurable 256 input axons and 256 output neurons. Input spikes enter the core through axons and the information is passed via the configurable binary synaptic crossbar to the leaky integrate-and-fire neurons, which act as computation units. The neuron assigns weights to the input spikes, integrates and updates its membrane potential, compares it to a threshold and may spike and reset, based on the selected threshold, reset, and stochastic modes [Cassidy et al., 2013].

Programming for TrueNorth means writing core configurations and connections, which in a way could be seen as building a neural network. A programming language has been developed to abstract the chip configuration [Amir et al., 2013]. In our case, the learned neural networks need to be translated into a valid TrueNorth configuration.

In previous work [Esser et al., 2015], a constrained network is trained, which learns the crossbar connection probability and biases. To deploy the model into TrueNorth these probabilities are sampled into discrete synaptic crossbar connections.

3 Constrained TrainingModel

Neuromorphic chips such as TrueNorth provide a reduction of power consumption, which is achieved by a constrained hardware architecture. There are several constraints compared to more traditional hardware platforms that are explored in this section to define the training of neuromorphic chips. These constraints are: use of spiking neurons, weights constrained to a limited set of integer values and reduced connectivity. In the following sections, we describe these constraints in more detail and provide a way to do learning based on backpropagation. We also describe how to use output layer predictions integrated into backpropagation.

3.1 Spiking Neurons

Spiking neurons have two possible outputs (0, no output, or 1, spike) as shown in the revised neuron output shown in eq. 4.

Spiking neurons behavior could be seen as a Heaviside step function, which does not have a gradient for backpropagation since it is not differentiable. This step function can have many different smooth approximations using, for instance, the logistic function that is used to represent the neuron output ˆnj as shown in eq. 5. The parameter k is the steepness of the function, which in our work has been set empirically to 1/2. Other values of k could be explored or, may be, approximate its value to a logistic distribution using large data sets.

The following gradient formulation can then be used in backpropagation.

In this approach, each neuron is stateless, so its status is reset after each time cycle, or tick. This is implemented in TrueNorth by setting parameters as specified and each neurons leak equal to the rounded bias term, with its threshold to 0, its membrane potential floor to 0 [Esser et al., 2015].

3.2 ConstrainedWeights

In TrueNorth, effective weights wij are divided in two components: the axon weight sij and the crossbar connection cij, so wij = cijsij. Taking this into account, the integration function is defined as:

Instead of learning the weights wij, the axon weights are proportionally distributed as explained below the binary crossbar connection cij is learnt during training.

The hardware accepts two possible values for the crossbar connections (0,1). The effective weights are dependent on the configuration of the axon types and crossbar connections. In this setup, the total number of effective weights that can be used is a maximum of five, i.e. four axon type weights and no connection (which means zero weight). The distribution of weights in sij depends on the axon type configuration in a TrueNorth core (we follow [Esser et al., 2015]).

The backpropagation gradients are specified as below and a similar formulation is used for bias values, which are round to the closer integer value during deployment. In the next section, we explain how the binary crossbar connections are learnt.

3.3 Binary Crossbar Connection Learning

In the previous section, we have shown that binary crossbar connections need to be learnt. We propose adapting previous work [Esser et al., 2016; Courbariaux et al., 2015] to learn binary crossbar connections that are used to obtain the effective network weights as shown in algorithm 1.

This is in contrast to previous work in which connection probabilities are learnt [Esser et al., 2015] that need to be sampled prior to hardware deployment. As well, in contrast to previous work [Esser et al., 2016; Courbariaux et al., 2015], the proposed approach allows learning a larger number of weights that are defined by the hardware configuration.

During training, in the forward propagation step the high precision csh matrix from the shadow network is used to calculate the c binary connection values (eq. 9), which are used to calculate the effective network weights using (wij = cijsij). Effective network weights are used to calculate the performance of the trained model (c.f. figure 3).

During backpropagation, gradients are estimated using c connection values that are used to update the csh high precision connection probabilities.

Algorithm 1 Forward and backward propagation

//Forward propagation for l = 1 : layers do for c = 1 : layer(l).cores do Calculate core binary crossbar connections cij (c.f. eq. 9) and effective weights wij Calculate core neuron outputs ˆ Ij using cij end for end for Estimate log loss in the last layer //Backpropagation for l = 1 : layers do for c = 1 : layer(l).cores do Calculate gradients using cij Update csh ij using estimated gradients end for end for

3.4 Neural Network Connections

The described method allows defining fully connected layer networks, but TrueNorth cores have 256 inputs (axons) and 256 outputs (neurons), which constrains the size of fully connected layers. When designing a network, the first layer, which processes the input data, is defined by blocks that tile to cover the input data. Block size and stride are defined by the number of pixels in the image classification case. Since each of these blocks maps to a TrueNorth core, the number of input axons required to map the input data per block is calculated as (blockSize2 ∗numberChannels) and the maximum value is 256, which is the number of axons per core. In the upper layers, block size and stride are defined by the number of cores (instead of image size as in the input layer). Output neurons of the previous layer are mapped to the input axons of the cores in the upper layer.

3.5 Output Layer

The last layer of the of the network is used for prediction. Spiking neurons in the last layer can be assigned randomly to one category [Esser et al., 2015]. The category with the highest number of neurons firing will be the predicted one.

These counts need to be used in backpropagation. In our work, we have used the number of spiking neurons per category csk divided by the number of neurons per category cnk and used softmax as shown in equation 10 to estimate the per category probability that is then used with the logarithmic loss (eq. 3).

We have described neural networks, the constraints added by neuromorphic chips and how to adapt the training of neural networks to these constraints. In the following sections, we present the results obtained using the MNIST data set and EEG data.

4 Results

We have used two data sets in our experiments. An EEG data set, described in more detail below, and the MNIST handwritten data set.

EEG data 1 was recorded from a participants wearing electrodes placed on the scalp while they performed a self-paced hand squeeze task. Full experimental details are described elsewhere [Nurse et al., 2015]. Briefly, the participant wore surface electromyography (EMG) electrodes attached to the forearms that were used to detect the onset of each hand squeeze. Data was sampled at 1 kHz in four minute sessions, with a total recording time of approximately 30 minutes. Detection of hand squeeze onset was achieved by extracting the root mean squared (RMS) power from the EMG signal, and the corresponding EEG data was labeled with either left-hand squeeze or right-hand squeeze (2 classes). Each EEG instance has been translated to 46x46 images with one 8-bit channel. In this study, EEG signals from participant D are used, which allows comparing the results to previous work. This data set contains 480/468 examples per class in the training set and 66/95 examples per class in the testing set. The data set consisted of a randomly allocated unseen test set (20% of the data) and training set (80% of the data), allowing a direct comparison with previous results [Nurse et al., 2015].

MNIST data set [LeCun et al., 1998] contains handwritten digits from 0 to 9 with 60,000 examples for training and 10,000 samples for testing. Images are 28x28 pixels, with one 8-bit gray scale channel. Gray scale values have been normalized dividing them by 255.

1 https://github.com/EwanNurse/A-Generalizable-BCI-usingMachine-Learning-for-Feature-Discovery

During training EEG and MNIST data are represented using floating point values. During deployment, the data values a single input axon over n ticks, which means that floating point values need to be represented as a set of spikes. Rate code [Jimeno Yepes et al., 2016] has been used to encode the data. In rate code, the number of spikes in the time window represents the encoded value, k. Spikes are evenly distributed within the time window.

Experiments were run using a MATLAB implementation of the proposed method, this includes training and evaluation of the trained models in a simulated MATLAB environment. We have used two configurations for the axon type weights in the neuron configuration s=[-1,1], used in previous work, and s=[-2,-1,1,2].

Table 1 lists the networks used in the experiments. Network configurations are derived from previous work [Esser et al., 2015]. There is a small network with 3 layers that requires 9 TrueNorth cores and a larger network with 4 layers that requires 30 TrueNorth cores.

Performance of the deployed trained models has been measured using classification accuracy, which is calculated by dividing the number of correctly predicted instances by all instances in the test set. McNemar's test has been used to look if the outcome of the systems is statistically significant [Haralick, 1993]. During test, data is converted into spikes using a rate code scheme. Since rate code is used and there is no need to sample from the trained network crossbar probabilities, results are stable (e.g. repetitions of the experiments provide the same result) compared to previous methods.

We have found that the initial optimal learning rate lr was provided by equation 11, which takes into account the number of neurons assigned to each category and the batch size.

4.1 EEGResults

For EEG experiments, the initial learning rate lr was multiplied by 0.1. We have updated the learning rate every 100 iterations using a rate of 0.1. Batch size is 25 and we have run a total of 500 epochs.

Ranges of values vary a lot for EEG data across features. We have normalized each individual feature xi but instead of using values between -1 and 1 as it is typically done with unconstrained neural networks [Ioffe and Szegedy, 2015], the values have been normalized to be between 0 and 1 as shown in the formula below. A small ϵ value has been inserted to avoid zero values.

Results are shown in Table 2 training the network with two different axon type weights and the combination of both trained models. Previously reported results using TrueNorth showed a 76% accuracy for participant D on this 2-class task [Nurse et al., 2016]. With the proposed method we obtain over 86%, which is the same accuracy as previously reported for the same data set using an unconstrained neural network [Nurse et al., 2015]. Better results are obtained when a more restricted set of axon type weights (s = [-1, 1]) is used. Normalization of the set at the feature level improves the performance of the trained models, which indicates that feature preprocessing can benefit predictive performance. Both results are statistically significant (p < 0.005). Using more than 32 ticks does not show a statistically significant improvement.

4.2 MNIST Results

Training has been carried out using batches of 100 images for each iteration and with and without data augmentation. To perform data augmentation, random changes to the images have been done in each epoch. We used a maximum rotation of 7.5 degrees, a maximum shift of 2.5 and a maximum rescale of 7.5%. Augmentation changes are randomly performed. For experiments without augmentation, the initial learning rate lr is multiplied by 0.1 and 500 epochs were used. For augmentation experiments, the initial learning rate lr is multiplied by 0.1, which is further multiplied by 0.1 after 2,000 epochs. 3,000 epochs were used for training. More iterations were required for the training set using augmentation to converge compared to the number of iterations used without augmentation.

Tables 3 and 4 show the results for several configurations. The large network shows better performance compared to the smaller one. The configuration with axon types (s = [-1, 1]) provide better performance. Augmentation increases the performance, which is similar to previous published work [Esser et al., 2015]. Improvements in performance by these three configurations are statistically significant (p < 0.005). As with the EEG results, using more than 32 ticks does not show a statistically significant improvement.

5 Discussion

Results on EEG data show that the performance of the trained network increased from 76% [Nurse et al., 2016] to 86%, which is close to state-of-the-art performance obtained using unconstrained fully connected layers [Nurse et al., 2015]. Results on MNIST show that the performance of the proposed method is similar compared to previously reported work depending on the configuration.

The use of several ticks for encoding input data shows that a plateau is reached after 8 ticks for MNIST, additional ticks to encode data does not significantly increase performance.

We have examined the effective weights and bias values. Figures 4 and 5 show the values for MNIST and figure 6 shows the values for the EEG data set.

In all combinations, approximately half of the weights have a value of zero for all the layers, which implies that half of the crossbar connections are disconnected. The other weights are evenly distributed. Furthermore, bias values tend to be around zero for all layers in many cases. On the other hand, when using augmentation in MNIST or in the EEG experiments, bias values tend to be distributed mostly on the negative values for the first layers. Negative values might indicate that there is a tendency to prevent neurons from spiking.

The values for weights and biases imply that in a deployment scenario, hardware configuration could be further constrained to a limited set of bias values and further research is required to understand implications in hardware design to improve crossbar usage.

Neuron weight configuration s=[-1,1]

6 Conclusions and FutureWork

We have shown that it is possible to train constrained fully connected layers on neuromorphic chips for two sets of problems (image classification with MNIST and EEG data analytics), which provides better or similar performance compared to previous work depending on the data set.

Results on EEG data sets which were previously analysed through the original method yield over 86% classification accuracy using the proposed approach, which is comparable to unconstrained neural networks. The trained models use a small portion of the TrueNorth chip (30 cores vs. 4096 available in the current version of the chip), thus requiring a much less than 70mW to work, which makes these models suitable for portable autonomous devices with large autonomy.

The outcome of this research provides insights into the possibility of learning network parameters that translate into deployment networks for brain-inspired chips. We will explore feasibility of expanding the introduced new methodology to train other kinds of existing deep learning algorithms including convolutional neural networks and recurrent neural networks. Furthermore, analysis of the learnt parameters provide insights that might complement hardware design, thus providing a more efficient deployment of the trained models.

Acknowledgments

The authors would like to thank the SyNAPSE team at the IBMAlmaden Research Center for their support.