{"type":"ScholarlyArticle","title":"Improving Classification Accuracy of Feedforward Neural Networks for Spiking Neuromorphic Chips","authors":[{"name":"Antonio Jimeno Yepes","slug":"antonio-jimeno-yepes","given":"Antonio Jimeno","family":"Yepes","affiliation":"IBM Research , VIC , Australia"},{"name":"Jianbin Tang","slug":"jianbin-tang","given":"Jianbin","family":"Tang","affiliation":"IBM Research , VIC , Australia"},{"name":"Benjamin Scott Mashford","slug":"benjamin-scott-mashford","given":"Benjamin Scott","family":"Mashford","affiliation":"IBM Research , VIC , Australia"}],"abstract":"Deep Neural Networks (DNN) achieve human level performance in many image analytics tasks but DNNs are mostly deployed to GPU platforms that consume a considerable amount of power. New hardware platforms using lower precision arithmetic achieve drastic reductions in power consumption. More recently, brain-inspired spiking neuromorphic chips have achieved even lower power consumption, on the order of milliwatts, while still offering real-time processing.\r\n\r\nHowever, for deploying DNNs to energy efficient neuromorphic chips the incompatibility between continuous neurons and synaptic weights of traditional DNNs, discrete spiking neurons and synapses of neuromorphic chips need to be overcome. Previous work has achieved this by training a network to learn continuous probabilities, before it is deployed to a neuromorphic architecture, such as IBM TrueNorth Neurosynaptic System, by random sampling these probabilities.\r\n\r\nThe main contribution of this paper is a new learning algorithm that learns a TrueNorth configuration ready for deployment. We achieve this by training directly a binary hardware crossbar that accommodates the TrueNorth axon configuration constrains and we propose a different neuron model.\r\n\r\nResults of our approach trained on electroencephalogram (EEG) data show a significant improvement with previous work (76% vs 86% accuracy) while maintaining state of the art performance on the MNIST handwritten data set.","doi":"10.24963/ijcai.2017/274","url":"https://ijcai.org/proceedings/2017/274","pdf":"https://ijcai.org/proceedings/2017/0274.pdf","markdown":"https://ijcai.org/proceedings/2017/274.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/274","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":274,"pages":{"first":1973,"last":1979},"keywords":["Machine Learning: Neural Networks","Machine Learning: Classification","Machine Learning: Machine Learning"],"videos":[],"references":[{"position":1,"raw":"[Amir et al., 2013] Arnon Amir, Pallab Datta, William P Risk, Andrew S Cassidy, Jeffrey Kusnitz, Steven K Esser, Alexander Andreopoulos, Theodore M Wong, Myron Flickner, Rodrigo Alvarez-Icaza, et al. Cognitive com- puting programming paradigm: a corelet language for Neuron weight configuration s=[-1,1]","title":"Cognitive computing programming paradigm: a corelet language for Neuron weight configuration","authors":"Arnon Amir, Pallab Datta, William P Risk, Andrew S Cassidy, Jeffrey Kusnitz, Steven K Esser, Alexander Andreopoulos, Theodore M Wong, Myron Flickner, Rodrigo Alvarez-Icaza","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"large-scale neural simulations. Proceedings of the IEEE, 102(5):699-716, 2014.","title":"large-scale neural simulations","authors":null,"year":"2014","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cassidy et al., 2013] Andrew Cassidy, Paul Merolla, John Arthur, Steve Esser, Bryan Jackson, Rodrigo Alvarez- Icaza, Pallab Datta, Jun Sawada, Theodore Wong, Vitaly Feldman, Arnon Amir, Daniel Ben-Dayan Rubin, et al. Cognitive computing building block: A versatile and effi- cient digital neuron model for neurosynaptic cores. In The 2013 International Joint Conference on Neural Networks (IJCNN), pages 1-10. IEEE, 2013.","title":"Cognitive computing building block: A versatile and efficient digital neuron model for neurosynaptic cores","authors":"Andrew Cassidy, Paul Merolla, John Arthur, Steve Esser, Bryan Jackson, Rodrigo Alvarez-Icaza, Pallab Datta, Jun Sawada, Theodore Wong, Vitaly Feldman, Arnon Amir, Daniel Ben-Dayan Rubin","year":"2013","venue":"The 2013 International Joint Conference on Neural Networks (IJCNN)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cheng et al., 2015] Zhiyong Cheng, Daniel Soudry, Zexi Mao, and Zhenzhong Lan. Training binary multilayer neural networks for image classification using expectation backpropagation. arXiv preprint arXiv:1503.03562, 2015.","title":"Training binary multilayer neural networks for image classification using expectation backpropagation","authors":"Zhiyong Cheng, Daniel Soudry, Zexi Mao, Zhenzhong Lan","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Courbariaux et al., 2014] Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Low precision arithmetic for deep learning. arXiv preprint arXiv:1412.7024, 2014.","title":"Low precision arithmetic for deep learning","authors":"Matthieu Courbariaux, Yoshua Bengio, Jean-Pierre David","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Courbariaux et al., 2015] Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propa- gations. In Advances in Neural Information Processing Systems, pages 3105-3113, 2015.","title":"Binaryconnect: Training deep neural networks with binary weights during propagations","authors":"Matthieu Courbariaux, Yoshua Bengio, Jean-Pierre David","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Duda et al., 2012] Richard O Duda, Peter E Hart, and David G Stork. Pattern classification. John Wiley & Sons, 2012.","title":"Pattern classification","authors":"Peter E Richard O Duda, David G Hart, Stork","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Esser et al., 2015] Steve K Esser, Rathinakumar Ap- puswamy, Paul Merolla, John V Arthur, and Dharmen- dra S Modha. Backpropagation for energy-efficient neuromorphic computing. In Advances in Neural Information Processing Systems, pages 1117-1125, 2015.","title":"Backpropagation for energy-efficient neuromorphic computing","authors":"Steve K Esser, Rathinakumar Appuswamy, Paul Merolla, John V Arthur, Dharmendra S Modha","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Esser et al., 2016] Steven K Esser, Paul A Merolla, John V Arthur, Andrew S Cassidy, Rathinakumar Appuswamy, Alexander Andreopoulos, David J Berg, Jeffrey L McK- instry, Timothy Melano, Davis R Barch, et al. Convo- lutional networks for fast, energy-efficient neuromorphic computing. arXiv preprint arXiv:1603.08270, 2016.","title":"Convolutional networks for fast, energy-efficient neuromorphic computing","authors":"Paul A Steven K Esser, John V Merolla, Andrew S Arthur, Rathinakumar Cassidy, Alexander Appuswamy, David J Andreopoulos, Jeffrey L Berg, Timothy Mckinstry, Melano, Davis R Barch","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Haralick, 1993] Robert M Haralick. Performance charac- terization in computer vision. In International Conference on Computer Analysis of Images and Patterns, pages 1-9. Springer, 1993.","title":"Performance characterization in computer vision","authors":"M Robert, Haralick","year":"1993","venue":"International Conference on Computer Analysis of Images and Patterns","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hwang and Sung, 2014] Kyuyeon Hwang and Wonyong Sung. Fixed-point feedforward deep neural network de- sign using weights+ 1, 0, and-1. In Signal Processing Sys- tems (SiPS), 2014 IEEE Workshop on, pages 1-6. IEEE, 2014.","title":"Fixed-point feedforward deep neural network design using weights+ 1, 0, and-1","authors":"Sung ; Hwang, Kyuyeon Hwang, Wonyong Sung","year":"2014","venue":"Signal Processing Systems (SiPS)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jimeno Yepes et al., 2016] Antonio Jimeno Yepes, Jianbin Tang, Shreya Saxena, Tobias Brosch, and Arnon Amir. Weighted population code for low power neuromorphic image classification. In Neural Networks (IJCNN), 2016 International Joint Conference on, pages 4294-4301. IEEE, 2016.","title":"Weighted population code for low power neuromorphic image classification","authors":"Jimeno Yepes","year":"2016","venue":"Neural Networks (IJCNN), 2016 International Joint Conference on","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lane et al., 2015] Nicholas D Lane, Sourav Bhattacharya, Petko Georgiev, Claudio Forlivesi, and Fahim Kawsar. An early resource characterization of deep learning on wear- ables, smartphones and internet-of-things devices. In Pro- ceedings of the 2015 International Workshop on Internet of Things towards Applications, pages 7-12. ACM, 2015.","title":"An early resource characterization of deep learning on wearables, smartphones and internet-of-things devices","authors":"Sourav Nicholas D Lane, Petko Bhattacharya, Claudio Georgiev, Fahim Forlivesi, Kawsar","year":"2015","venue":"Proceedings of the 2015 International Workshop on Internet of Things towards Applications","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[LeCun et al., 1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning ap- plied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Yann Lecun, Léon Bottou, Yoshua Bengio, Patrick Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Merolla et al., 2014] Paul A Merolla, John V Arthur, Ro- drigo Alvarez-Icaza, Andrew S Cassidy, Jun Sawada, Fil- ipp Akopyan, Bryan L Jackson, Nabil Imam, Chen Guo, Yutaka Nakamura, et al. A million spiking-neuron inte- grated circuit with a scalable communication network and interface. Science, 345(6197):668-673, 2014.","title":"A million spiking-neuron integrated circuit with a scalable communication network and interface","authors":"John V Paul A Merolla, Rodrigo Arthur, Andrew S Alvarez-Icaza, Jun Cassidy, Filipp Sawada, Akopyan, Nabil Bryan L Jackson, Chen Imam, Yutaka Guo, Nakamura","year":"2014","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Nurse et al., 2015] Ewan S Nurse, Philippa J Karoly, David B Grayden, and Dean R Freestone. A generalizable brain-computer interface (bci) using machine learning for feature discovery. PloS one, 10(6):e0131328, 2015.","title":"A generalizable brain-computer interface (bci) using machine learning for feature discovery","authors":"Philippa J Ewan S Nurse, David B Karoly, Dean R Grayden, Freestone","year":"2015","venue":"PloS one","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Nurse et al., 2016] Ewan Nurse, Benjamin S Mashford, An- tonio Jimeno Yepes, Isabell Kiral-Kornek, Stefan Harrer, and Dean R Freestone. Decoding eeg and lfp signals using deep learning: heading truenorth. In Proceedings of the ACM International Conference on Computing Frontiers, pages 259-266. ACM, 2016.","title":"Decoding eeg and lfp signals using deep learning: heading truenorth","authors":"Nurse","year":"2016","venue":"Proceedings of the ACM International Conference on Computing Frontiers","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Painkras et al., 2013] Eustace Painkras, Luis A Plana, Jim Garside, Steve Temple, Francesco Galluppi, Cameron Pat- terson, David R Lester, Andrew D Brown, and Steve B Furber. Spinnaker: A 1-w 18-core system-on-chip for massively-parallel neural network simulation. IEEE Jour- nal of Solid-State Circuits, 48(8):1943-1953, 2013.","title":"Spinnaker: A 1-w 18-core system-on-chip for massively-parallel neural network simulation","authors":"Eustace Painkras, Luis A Plana, Jim Garside, Steve Temple, Francesco Galluppi, Cameron Patterson, David R Lester, Andrew D Brown, Steve B Furber","year":"2013","venue":"IEEE Journal of Solid-State Circuits","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pfeil et al., 2012] Thomas Pfeil, Andreas Grübl, Sebastian Jeltsch, Eric Müller, Paul Müller, Mihai A Petrovici, Michael Schmuker, Daniel Brüderle, Johannes Schem- mel, and Karlheinz Meier. Six networks on a univer- sal neuromorphic computing substrate. arXiv preprint arXiv:1210.7083, 2012.","title":"Six networks on a universal neuromorphic computing substrate","authors":"Thomas Pfeil, Andreas Grübl, Sebastian Jeltsch, Eric Müller, Paul Müller, A Mihai, Michael Petrovici, Daniel Schmuker, Johannes Brüderle, Karlheinz Schemmel, Meier","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Poggi and Mattoccia, 2016] Matteo Poggi and Stefano Mat- toccia. A wearable mobility aid for the visually impaired based on embedded 3d vision and deep learning. In Com- puters and Communication (ISCC), 2016 IEEE Sympo- sium on, pages 208-213. IEEE, 2016.","title":"A wearable mobility aid for the visually impaired based on embedded 3d vision and deep learning","authors":"Matteo Poggi, Stefano Mattoccia","year":"2016","venue":"Computers and Communication (ISCC)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Stromatias et al., 2015] E Stromatias, D Neil, F Galluppi, M Pfeiffer, SC Liu, and S Furber. Scalable energy- efficient, low-latency implementations of spiking deep belief networks on spinnaker. In 2015 International Joint Conference on Neural Networks (IJCNN)(Killarney), 2015.","title":"Scalable energyefficient, low-latency implementations of spiking deep belief networks on spinnaker","authors":"Stromatias, F Neil, M Galluppi, Pfeiffer, Liu, Furber","year":"2015","venue":"2015 International Joint Conference on Neural Networks (IJCNN)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}