{"type":"ScholarlyArticle","title":"Diverse Neuron Type Selection for Convolutional Neural Networks","authors":[{"name":"Guibo Zhu","slug":"guibo-zhu","given":"Guibo","family":"Zhu","affiliation":"Research Center for Brain-inspired Intelligence , Institute of Automation , Chinese Academy of Sciences (CASIA); National Laboratory of Pattern Recognition , CASIA"},{"name":"Zhaoxiang Zhang","slug":"zhaoxiang-zhang","given":"Zhaoxiang","family":"Zhang","affiliation":"Research Center for Brain-inspired Intelligence , Institute of Automation , Chinese Academy of Sciences (CASIA); National Laboratory of Pattern Recognition , CASIA; Center for Excellence in Brain Science and Intelligence Technology , CAS; University of Chinese Academy of Sciences"},{"name":"Xu-Yao Zhang","slug":"xu-yao-zhang","given":"Xu-Yao","family":"Zhang","affiliation":"National Laboratory of Pattern Recognition , CASIA"},{"name":"Cheng-Lin Liu","slug":"cheng-lin-liu","given":"Cheng-Lin","family":"Liu","affiliation":"National Laboratory of Pattern Recognition , CASIA; Center for Excellence in Brain Science and Intelligence Technology , CAS; University of Chinese Academy of Sciences"}],"abstract":"The activation function for neurons is a prominent element in the deep learning architecture for obtaining high performance. Inspired by neuroscience findings, we introduce and define two types of neurons with different activation functions for artificial neural networks: excitatory and inhibitory neurons, which can be adaptively selected by self-learning. Based on the definition of  neurons, in the paper we not only unify the mainstream activation functions, but also discuss the complementariness among these types of neurons. In addition, through the cooperation of excitatory  and inhibitory neurons, we present a compositional activation function that leads to new state-of-the-art performance comparing to rectifier linear units. Finally, we hope that our framework  not only gives a basic unified framework of the existing activation neurons to  provide guidance for future design, but also contributes neurobiological explanations which can be treated as a window to bridge the gap between biology and computer science.","doi":"10.24963/ijcai.2017/498","url":"https://ijcai.org/proceedings/2017/498","pdf":"https://ijcai.org/proceedings/2017/0498.pdf","markdown":"https://ijcai.org/proceedings/2017/498.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/498","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning S-Z","track":"Main track","number":498,"pages":{"first":3560,"last":3566},"keywords":["Machine Learning: Deep Learning","Machine Learning: Neural Networks"],"videos":[],"references":[{"position":1,"raw":"[Agostinelli et al., 2014] Forest Agostinelli, Matthew Hoff- man, Peter Sadowski, and Pierre Baldi. Learning activation functions to improve deep neural networks. arXiv:1412.6830, 2014.","title":"Learning activation functions to improve deep neural networks","authors":"Agostinelli","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio, 2009] Yoshua Bengio. Learning deep architectures for ai. Foundations and trends R ⃝ in Machine Learning, 2(1):1-127, 2009.","title":"Learning deep architectures for ai","authors":"Yoshua Bengio","year":"2009","venue":"Foundations and trends R ⃝ in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Clevert et al., 2015] Djork-Arné Clevert, Thomas Un- terthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv:1511.07289, 2015.","title":"Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units","authors":"Clevert","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dayan and Abbott, 2001] Peter Dayan and Laurence F Ab- bott. Theoretical neuroscience, volume 10. Cambridge, MA: MIT Press, 2001.","title":"Peter Dayan and Laurence F Abbott","authors":"Abbott Dayan","year":"2001","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248-255. IEEE, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Girshick et al., 2014] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, pages 580-587, 2014.","title":"Rich feature hierarchies for accurate object detection and semantic segmentation","authors":"Ross Girshick, Jeff Donahue, Trevor Darrell, Jitendra Malik","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Glorot et al., 2011] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Aistats, volume 15, page 275, 2011.","title":"Deep sparse rectifier neural networks","authors":"Xavier Glorot, Antoine Bordes, Yoshua Bengio","year":"2011","venue":"Aistats","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Goodfellow et al., 2013] Ian J Goodfellow, David Warde- Farley, Mehdi Mirza, Aaron C Courville, and Yoshua Ben- gio. Maxout networks. ICML (3), 28:1319-1327, 2013.","title":"Maxout networks","authors":"Ian J Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron C Courville, Yoshua Bengio","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpass- ing human-level performance on imagenet classification. In ICCV, pages 1026-1034, 2015.","title":"Delving deep into rectifiers: Surpassing human-level performance on imagenet classification","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hinton et al., 1986] Geoffrey E Hinton, James L Mcclel- land, and David E Rumelhart. Distributed representa- tions, parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations, 1986.","title":"Distributed representations, parallel distributed processing: explorations in the microstructure of cognition","authors":"Geoffrey E Hinton, James L Mcclelland, David E Rumelhart","year":"1986","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv:1502.03167, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Sergey Ioffe, Christian Szegedy","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jin et al., 2015] Xiaojie Jin, Chunyan Xu, Jiashi Feng, Yun- chao Wei, Junjun Xiong, and Shuicheng Yan. Deep learning with s-shaped rectified linear activation units. arXiv:1512.07030, 2015.","title":"Deep learning with s-shaped rectified linear activation units","authors":"Xiaojie Jin, Chunyan Xu, Jiashi Feng, Yunchao Wei, Junjun Xiong, Shuicheng Yan","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Krizhevsky and Hinton, 2009] Alex Krizhevsky and Geof- frey Hinton. Learning multiple layers of features from tiny images. 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, pages 1097- 1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kvitsiani et al., 2013] D Kvitsiani, S Ranade, B Hangya, H Taniguchi, JZ Huang, and A Kepecs. Distinct be- havioural and network correlates of two interneuron types in prefrontal cortex. Nature, 498(7454):363-366, 2013.","title":"Distinct behavioural and network correlates of two interneuron types in prefrontal cortex","authors":"Kvitsiani, B Ranade, H Hangya, Taniguchi, Huang, Kepecs","year":"2013","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2016] Yang Li, Chunxiao Fan, Yong Li, and Qiong Wu. Improving deep neural network with multiple para- metric exponential linear units. arXiv:1606.00305, 2016.","title":"Improving deep neural network with multiple parametric exponential linear units","authors":"Yang Li, Chunxiao Fan, Yong Li, Qiong Wu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liao et al., 2016] Minghui Liao, Baoguang Shi, Xiang Bai, Xinggang Wang, and Wenyu Liu. Textboxes: A fast text detector with a single deep neural network. arXiv preprint arXiv:1611.06779, 2016.","title":"Textboxes: A fast text detector with a single deep neural network","authors":"Liao","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lin et al., 2013] Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. arXiv:1312.4400, 2013.","title":"Network in network","authors":"Lin","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Maas et al., 2013] Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. Rectifier nonlinearities improve neural net- work acoustic models. In Proc. ICML, volume 30, 2013.","title":"Rectifier nonlinearities improve neural network acoustic models","authors":"Awni Y Andrew L Maas, Andrew Y Hannun, Ng","year":"2013","venue":"Proc. ICML","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Markram et al., 2004] Henry Markram, Maria Toledo- Rodriguez, Yun Wang, Anirudh Gupta, Gilad Silberberg, and Caizhi Wu. Interneurons of the neocortical inhibitory system. Nature Reviews Neuroscience, 5(10):793-807, 2004.","title":"Interneurons of the neocortical inhibitory system","authors":"Henry Markram, Maria Toledo-Rodriguez, Yun Wang, Anirudh Gupta, Gilad Silberberg, Caizhi Wu","year":"2004","venue":"Nature Reviews Neuroscience","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Nair and Hinton, 2010] Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann ma- chines. In ICML, pages 807-814, 2010.","title":"Rectified linear units improve restricted boltzmann machines","authors":"Vinod Nair, Geoffrey E Hinton","year":"2010","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Pi et al., 2013] Hyun-Jae Pi, Balázs Hangya, Duda Kvit- siani, Joshua I Sanders, Z Josh Huang, and Adam Kepecs. Cortical interneurons that specialize in disinhibitory con- trol. Nature, 503(7477):521-524, 2013.","title":"Cortical interneurons that specialize in disinhibitory control","authors":"Hyun-Jae Pi, Balázs Hangya, Duda Kvitsiani, Joshua I Sanders, Josh Huang, Adam Kepecs","year":"2013","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Shah et al., 2016] Anish Shah, Eashan Kadam, Hena Shah, and Sameer Shinde. Deep residual networks with expo- nential linear unit. arXiv:1604.04112, 2016.","title":"Deep residual networks with exponential linear unit","authors":"Anish Shah, Eashan Kadam, Hena Shah, Sameer Shinde","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Shang et al., 2016] Wenling Shang, Kihyuk Sohn, Diogo Almeida, and Honglak Lee. Understanding and improving convolutional neural networks via concatenated rectified linear units. arXiv:1603.05201, 2016.","title":"Understanding and improving convolutional neural networks via concatenated rectified linear units","authors":"Wenling Shang, Kihyuk Sohn, Diogo Almeida, Honglak Lee","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Shi et al., 2016] Baoguang Shi, Xiang Bai, and Cong Yao. An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016.","title":"An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition","authors":"Shi","year":"2016","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Simonyan and Zisserman, 2014a] Karen Simonyan and An- drew Zisserman. Two-stream convolutional networks for action recognition in videos. In Advances in Neural Infor- mation Processing Systems, pages 568-576, 2014.","title":"Two-stream convolutional networks for action recognition in videos","authors":"Zisserman ; Simonyan, Zisserman","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Simonyan and Zisserman, 2014b] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556, 2014.","title":"Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition","authors":"Zisserman Simonyan","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Trottier et al., 2016] Ludovic Trottier, Philippe Giguère, and Brahim Chaib-draa. Parametric exponential linear unit for deep convolutional neural networks. arXiv:1605.09332, 2016.","title":"Parametric exponential linear unit for deep convolutional neural networks","authors":"Ludovic Trottier, Philippe Giguère, Brahim Chaib-Draa","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[van Vreeswijk and Sompolinsky, 1996] Carl van Vreeswijk and Haim Sompolinsky. Chaos in neuronal networks with balanced excitatory and inhibitory activity. Science, 274(5293):1724, 1996.","title":"Chaos in neuronal networks with balanced excitatory and inhibitory activity","authors":"Sompolinsky ; Carl Van Vreeswijk, Haim Van Vreeswijk, Sompolinsky","year":"1996","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Xu et al., 2015] Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in con- volutional network. arXiv:1505.00853, 2015.","title":"Empirical evaluation of rectified activations in convolutional network","authors":"Bing Xu, Naiyan Wang, Tianqi Chen, Mu Li","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zagoruyko and Komodakis, 2016] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv:1605.07146, 2016.","title":"Wide residual networks","authors":"Sergey Zagoruyko, Nikos Komodakis","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2016] Chao Zhang, Junchi Yan, Changsheng Li, Xiaoguang Rui, Liang Liu, and Rongfang Bie. On estimating air pollution from photos using convolutional neural network. In MM, pages 297-301. ACM, 2016.","title":"On estimating air pollution from photos using convolutional neural network","authors":"Chao Zhang, Junchi Yan, Changsheng Li, Xiaoguang Rui, Liang Liu, Rongfang Bie","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}