Diverse Neuron Type Selection for Convolutional Neural Networks

1 Introduction

Convolutional neural networks (CNNs) [Krizhevsky et al., 2012] have recently been widely used in different computer vision tasks. Deep CNNs not only significantly improve the image recognition accuracy [Krizhevsky et al., 2012; Simonyan and Zisserman, 2014b], but also play important roles in variety of vision tasks such as object detection [Girshick et al., 2014], text detection [Shi et al., 2016; Liao et al., 2016], human activity recognition [Simonyan and Zisserman, 2014a], and air pollution [Zhang et al., 2016]. To improve the performance of CNNs, there are so many techniques advancing in CNNs performance improvements which can be roughly divided into network structure design (e.g. , depth [Simonyan and Zisserman, 2014b], modified nonlinear activation functions [Nair and Hinton, 2010]) and overfitting preventing techniques(e.g. , normalization [Ioffe and Szegedy, 2015]).

∗Corresponding author.

Among these techniques, one key factor leading to the recent success of deep neural networks is the invention of the rectifier functions to model the neuron activation, i.e. Rectified Linear Unit (ReLU) [Nair and Hinton, 2010; Glorot et al., 2011]. It accelerates the learning process [Nair and Hinton, 2010] and significantly improves the overall network performance compared with traditional sigmoid or tanh units [Glorot et al., 2011] by suppressing the vanishing/exploding gradient problem in training deep networks. Due to the importance of the activation function, many researchers have proposed elaborate activation functions for improving the network performance. These works can be coarsely summarized as two categories: (1) modified ReLUs (e.g. , Leaky ReLU (LReLU) [Maas et al., 2013], Parametric ReLU [He et al., 2015], Random ReLU (RReLU) [Xu et al., 2015], Exponential Linear Units (ELU) [Clevert et al., 2015], Parametric ELU (PELU) [Trottier et al., 2016], Multiple Parametric Exponential Linear Units (MPELU) [Li et al., 2016]) which remove zero gradients with various functions in negative input, and (2) ensemble activation functions (e.g. , Maxout [Goodfellow et al., 2013], Adaptive Piecewise Linear (APL) [Agostinelli et al., 2014], Network in Network [Lin et al., 2013], S-shaped ReLU (SReLU) [Jin et al., 2015], Concatenated ReLU (CReLU) [Shang et al., 2016]) which take advantage of the ensemble power.

Despite the prevalence of the activation functions, there are rarely proposals paying attention to their common properties in the perspective of neurological evidence. In the neuroscience domain, there are two main types of neurons in cerebral cortex (i.e. , the surface of the brain): the primary neuron is excitatory neuron and the relay neuron is inhibitory neuron [Markram et al., 2004]. Under normal circumstances, two types of cells regulate themselves each other to keep brain in dynamic equilibrium of excitement and inhibition state [Pi et al., 2013]. Therefore, they both play important roles in the neural network of brain. Inspired by the neurological knowledge, in this paper we propose two types of neurons with different activation functions for artificial neural networks: excitatory and inhibitory neurons, which are learned for the neuron type selection. Based on the proposed neurons, we not only unify the mainstream activation functions, but also discuss the complementariness among these neuron types in the view of neuroscience. Additionally, through the cooperation of excitatory and inhibitory neurons, experiments demonstrate that the proposed activation functions work well for training neural networks and achieve state-of-the-art performance on several image recognition benchmarks.

2 RelatedWorks

We review the existing works related to activation functions, which are the focus of our research in this paper. They can mainly be divided into two categories: rectified unit family and ensemble activation function.

2.1 Rectified Unit Family

Since activation functions help to bring non-linearity, they play an important role in deep neural networks for their significant effect on the overall performance. ReLU [Nair and Hinton, 2010] used linear activation function to replace the saturated counterpart of the convolutional sigmoid or tanhlike units for positive inputs and set the output of negative inputs as zeros, i.e. , f(x) = max(0, x). It alleviated the exploding/vanishing gradient problem and expedited the convergence rate. Different from ReLU, LReLU [Maas et al., 2013], PReLU and RReLU incorporated a positive slope parameter for negative input in activation units to remove zero gradients. They are different from each other in the acquisition methods for the slope parameter, where LReLU is pre-defined before the training process, PReLU is adaptively learned and RReLU is randomly sampling based on stochastic regularization in the training process. ELU [Clevert et al., 2015] was defined as an identify function for positive inputs and exp(x)-1 for negative ones, which can handle the largen variance and bias shift problems by allowing the mean activations closer to zeros. Similar to PReLU, MPELU [Li et al., 2016] introduced a non-zero slope parameter for the negative parts of ELU and proposed a weight initialization strategy for ELU series. PELU [Trottier et al., 2016] introduced several slope parameters in positive/negative parts and adjusted them in training by modifying ELU with differentiability near zero.

2.2 Ensemble Activation Function

Ensemble activation functions have powerful approximate ability for any smooth functions in theory. Maxout [Goodfellow et al., 2013] used the maxout activation unit to output the maximum over several learned transformation functions for each neuron input. Adaptive Piecewise Linear (APL) [Agostinelli et al., 2014] was a weight sum of parameterized hinge-shaped functions whose parameters are learned during training. Network in Network [Lin et al., 2013] approximated the nonlinear rectified activation function in convolution networks with a multilayer perceptron in which the parameters are learned from data. S-shaped Rectified Linear Activation Units (SReLU) [Jin et al., 2015] formulated three piecewise linear functions with four learnable parameters which can imitate complex convex and non-convex functions in psychophysics and neural sciences. With the observed property of filters with opposite phase in lower layers, Concatenated ReLU (CReLU) [Shang et al., 2016] was designed by making an identical copy of the output values of the convolution layer, negating them, concatenating both the negated part and the original one, and then applying ReLU altogether.

3 Excitatory and Inhibitory Activation Function

In this section, we first give the definitions of excitatory and inhibitory neurons for artificial neural networks. Then we propose the compositional activation function of the two neurons. Finally, we discuss the intrinsic relationships among the mainstream activation functions and give some explanations.

3.1 Two Types of Neurons

Inspired by the neuroscience [Markram et al., 2004], we can define two types of neurons for artificial neural networks. Not every signal is useful for a particular task in human decisionmaking, therefore, there exit two types of different neurons. The first is "excitatory neuron" which is a function preserving the input signal well. The second is "inhibitory neuron" that will suppress the input signal. In this way, the neural networks can select the signals dynamically according to the requirement of the task in a data-driven manner. In previous researches, most widely-used neurons are excitatory, and the inhibitory neuron is not well-studied in the literature. In this paper, we will show that both of these two types of neurons are important and complementary for neural networks.

Actually, the excitatory and inhibitory neurons can be defined in many different formulations. In this paper, inspired from the Occam's razor theory (i.e., entities should not be multiplied unnecessarily), we give simple and straightforward definitions for them. Excitatory neuron: The neuron can be defined as:

which means the output f(x) preserves the absolute value of the input x without suppression under some conditions.

For an excitatory neuron, the simplest linear activation function is f(x) = x. However, it can not make an artificial neural network possess the ability of nonlinear mapping property. For better nonlinear data fitting, the simplest non-linear activation function is f(x) = max(x, 0), i.e. , ReLU [Nair and Hinton, 2010]. Because it satisfies the condition of Eq. (1), we think it is a representative case of excitatory neuron. Since there are more than one excitatory neurons in neural networks, here we only present four excitatory activation neurons in different quadrants as shown in Fig. 1. They are: f1(x), f2(x), f3(x) and f4(x) where f1(x) = max(x, 0) in the first quadrant as shown in Fig. 1(b), f2(x) = max(-x, 0) in the second quadrant as shown in Fig. 1(a), f3(x) = -max(-x, 0) in the third quadrant Fig. 1(c) and f4(x) = -max(x, 0) in the forth quadrant as shown in Fig. 1(d), respectively. These activation neurons are excitatory because they all satisfy Eq. (1).

Inhibitory Neuron: Not only are the excitatory neurons important, but also the inhibitory neurons play a key role in neural network [van Vreeswijk and Sompolinsky, 1996]. With the inhibitory neurons, one neural network can keep in equilibrium by restraining various neuron responses in different layers. Here we also give the definition of the inhibitory neuron as:

which means that the output of inhibitory neuron f(x) does not exceed a "constant" or tend towards stability even though the absolute value of the input signal x tends to infinity.

There are many inhibitory neurons with different activation functions. To simulate the inhibitory activation neurons which restrain the input signal, in this paper we just introduce four inhibitory activation functions in each quadrant as shown in Fig. 2 corresponding to the excitatory neurons. They are: f5(x), f6(x), f7(x) and f8(x) where f5(x) = tanh(max(x, 0) in the first quadrant as shown in Fig. 2(b), f6(x) = tanh(max(-x, 0)) in the second quadrant as shown in Fig. 2(a), f7(x) = exp(-max(-x, 0)) - 1 in the third quadrant as shown in Fig. 2(c) and f8(x) = exp(-max(x, 0)) - 1 as shown in Fig. 2(d), respectively.

3.2 Composition of Excitatory and Inhibitory Neurons

Excitatory neuron preserves the input signal while the inhibitory neuron suppresses the signal. They play different roles in neural networks. There are three most possible compositions among excitatory and inhibitory neurons: (1) excitatory and excitatory neurons; (2) excitatory and inhibitory neurons; (3) inhibitory and inhibitory neurons.

Excitatory and Excitatory Neurons: CReLU [Shang et al., 2016] is the representative one which is concatenated by the excitatory neurons or activation functions with Fig. 1(a) and Fig. 1(b) in parallel. Meanwhile, the importance of the ways to combine excitatory neurons is different. Specially, not any two excitatory neurons to combine is a good choice because of their relevance. If they are independent, they help each other; otherwise, the coupling relationship between themmay lead to the oscillation of the learning process, e.g. , the composition of Fig. 1(b) and Fig. 1 (d).

Excitatory and Inhibitory Neurons: We need to explore which composition between excitatory activation function and inhibitory activation function has the best complementariness. After investigating the well-known rectifiers, e.g. , LReLU [Maas et al., 2013], PReLU [He et al., 2015], ELU [Clevert et al., 2015] and PELU [Trottier et al., 2016], we argue that the parallel composition of the excitatory neuron in Fig. 1 (b) and the inhibitory neuron in Fig. 2 (c) has better complementariness with each other.

Following the complementary direction of neuron compositions, we design an excitatory and inhibitory activation (EIA) function with different neurons as shown in Fig. 3, represented by δn : Θ → Θ2 activation as follows: ∀x ∈ Θ, δn ≜ (e1(x), e2(x)). Here, e1(x) = max(x, 0), e2(x) = αexp(min( x β , 0)) - 1. The optimization algorithm of the parameters α and β is back-propagation similar to [Trottier et al., 2016]. The motivation of our EIA method is not only to reduce the bias shift of layer-wise convolution, but also keep the filter to be activated in an adaptive way. (e.g. , in the positive and negative direction or the antipodal inhibited way).

Inhibitory and Inhibitory Neurons: There exists the composition of inhibitory and inhibitory neurons [Kvitsiani et al., 2013]. Tanh [Hinton et al., 1986], i.e. , f(x) = e2x-1 e2x+1 , is the addition of two inhibitory neurons f5(x) and f6(x).

�

�

Neurological Guess: The proportion of inhibitory neurons in total neurons is almost 10% [Markram et al., 2004] while the other is mainly excitatory neurons. Suppose the negative input needs to be suppressed by inhibitory neurons and the positive output transmitted by the excitatory activation, the weight ratio between them should be 1 : 9. Given an input signal x after preprocessing of batch normalization [Ioffe and Szegedy, 2015], it satisfies the normal gaussian distribution x ∼ N(µ, δ2 ). We introduce a general activation function:

According to the ratio of excitatory activation and inhibitory activation, we guess there exists the expression as follows,

where ∫ is the integral operator, | · | gets the absolute value and p(x) is the probability density function. According to Eq. (4) and numerical integration algorithm, given g1(x) = x and g2(x) = αx, the parameter α is optimally initialized as 19 .

To make the excitatory neurons play a leading role, we need to penalize the inhibitory neurons with adaptive slope. In the literature, to handle the zero gradient problem during optimization, Maas et al. [Maas et al., 2013] proposed the non-zero slope for the activation function of the negative input counterpart which is potentially more robust. Inspired by PReLU [He et al., 2015], ELU [Clevert et al., 2015] and PELU [Trottier et al., 2016], the parameter with adaptive learning is a good strategy which reduces the overfitting risk of human interventions and alleviates the dynamic gradient propagation. More punishment reduces the noise effect of smaller negative input. In addition, the adaptive slope can be learned itself through the end-to-end network training.

Fractal Assumption: Divide an activation function into different quadrants and then conquer them as shown in Fig. 4, which are complementary.

The objectives of deep neural networks [Bengio, 2009; Glorot et al., 2011] is learning to disentangle and represent the factors of variation in the data. While concatenating with independent and appropriate quadrant functions, there are many advantages of sparsity (e.g. , information disentangling, efficient variable-size representation, linear separability and distributed but sparse) [Glorot et al., 2011] for the deep neural networks. The importance of different quadrants is statistically mining the collective knowledge in the literature and our empirical experiments.

3.3 Neurological Explantation

We will introduce the advantages of excitatory and inhibitory neurons from three aspects: using only one type of excitatory neuron, the collaboration of two types of excitatory neurons and the complementariness of excitatory and inhibitory neurons. Since excitatory neurons take a dominant role for neuroscience, only one type of excitatory neuron can still work very well in the neural networks for multiple tasks, such as the most widely used ReLU activation function [Nair and Hinton, 2010]. On the other hand, there are a variety of neurons which can construct different activation functions. For example, since CReLU [Shang et al., 2016] is the collaboration of two types of excitatory neurons, its performance is satisfactory. In fact, we know that there are also many inhibitory neurons in the neural networks from neurological knowledge. Although the excitatory neuron is very important, it is better to add inhibitory neurons into excitatory neurons for the highlevel semantic cognition task where the different neurons are complementary. Therefore, we propose the compositional excitatory and inhibitory activation (EIA) function which leads to much better performance in the following experiments.

4 Experiments

To evaluate the effects of our proposed activation functions, we conducted experiments on three datasets including CIFAR-10 and CIFAR-100 [Krizhevsky and Hinton, 2009] with the most common used deep neural network VGG [Simonyan and Zisserman, 2014b] and the ImageNet benchmark dataset [Deng et al., 2009] with the AlexNet [Krizhevsky et al., 2012] network architecture. The adopted VGG was modified with 19 layers and batch normalization for implementing on CIFAR-10/100. We chose VGG and CIFAR-10/100 to perform an in-depth investigation because the computational cost of a large network on the datasets is moderate and about ten hours on a K40 GPU. Note that the implementation of the VGG architecture was following to [Zagoruyko and Komodakis, 2016] who had released their source code. Except only changing the types of the activation functions for fair comparison, we kept all the hyperparameters remaining the same for all experiments. Additionally, we selected AlexNet with batch normalization using Torch for evaluating our proposed EIA activation function in ImageNet.

4.1 Datasets

The CIFAR-10/100 datasets [Krizhevsky and Hinton, 2009] contain 50, 000 training and 10, 000 testing samples with size of 32×32 color images from 10 and 100 classes, respectively. The data preprocessing techniques follow the method used in [Goodfellow et al., 2013] ImageNet includes 1.3M training and 100K testing images for 1000 categories with one crop.

4.2 Evaluation on Single Neuron Type

To evaluate different kinds of excitatory neuron types, we designed several experiments with these neurons. First, we tested four types of excitatory neurons. ReLU [Nair and Hinton, 2010] was treated as the first type of excitatory neuron E1. E2, E3, and E4 were the remaining types of excitatory neurons corresponding to the activation functions f2(x), f3(x) and f4(x) respectively. Similarly, four types of inhibitory neurons (i.e. , I1, I2, I3, and I4 corresponding to activation functions f5(x), f6(x), f7(x) and f8(x) were also compared. However, these inhibitory neurons were not convergent with the original VGG model. Even with batch normalization, their performance still had significant difference and worse than the excitatory neurons. As shown in Table 1, the first type of excitatory neuron achieved the best performance in these neuron types.

4.3 Evaluation on Coupling Excitatory Neurons

To investigate the cooperation with excitatory and excitatory neurons, we compared various compositions with different types of excitatory neurons. These compositions are as follows: AVR is f(x) = f1(x) + f2(x), EE-1-2 is f(x) = (f1(x), f2(x)), EE-1-3 is f(x) = (f1(x), f3(x)), EE-1-4 is f(x) = (f1(x), f4(x)), EE-2-3 is f(x) = (f2(x), f3(x)), EE-2-4 is f(x) = (f2(x), f4(x)) and EE-3-4 is f(x) = (f3(x), f4(x)). As shown in Table 2, we found that the performance of compositions with Index 2-7 was all better than AVR which meant that the concatenated structure in parallel was better. That was because the concatenated structure in parallel not only utilized the advantage of sparsity, but also explored the complementary information among independent quadrants which consisted of the phase information. While evaluating the compositions with Index 2-7, we found that concatenations with independent quadrant functions in the diagonal line (e.g. , EE-1-2, EE-1-3, EE-3-4) were better than the coupling quadrant functions at only one side of axis (e.g.

, EE-2-3 and EE-1-4). Additionally, the best composition excitatory and excitatory neurons was better than the best type of neuron in Table 1 which showed their collaboration.

4.4 Evaluation on Pairs of One Excitatory Neuron and One Inhibitory Neuron

To evaluate the complementariness of excitatory and inhibitory neurons, we conducted several compositions among them. They were EI-1-3, EI-2-4 and EIA where EI-1-3 was constructed by the first type of excitatory neuron f1(x) and the third inhibitory neuron f7(x), EI-2-4 was f(x) = (f2(x), f8(x)) and EIA was our proposed activation function. The main difference between EI-1-3 and EIA was that we penalized the inhibitory neurons with adaptive slopes from self-learning. Table 3 showed that EIA got the best performance which verified the importance of diverse neuron type selection coming from self-learning in the end-to-end convolutional neural network structure.

Besides, we also compared paired inhibitory and inhibitory neurons with EIA. Their performance was also not as good as EIA. For simplicity, we did not give the results here.

4.5 Evaluation of Fractal Assumption

In commercial applications, we need to consider the limitation of device memory and time complexity. With the fractal assumption of different excitatory and inhibitory neurons, we can select various neurons to design new activation functions. For example, AVR can be constructed by two excitatory neurons, i.e. , f1(x) and f2(x), Linear can be generated by f1(x) and f3(x) and ELU can be divided into one excitatory neuron f1(x) and one inhibitory neuron f7(x). There are the fractal and concatenated structures among them. Specifically, AVR, f(x) = max(x, 0)+max(-x, 0), equals to the inverse composition result of EE-1-2. There was the same relation between Linear, f(x) = max(x, 0) + min(x, 0), and EE1-3, f(x) = (max(x, 0),min(x, 0)). As show in Table 4, we found that the fractal activation functions (i.e. , CReLU and EE-1-3) had a significant improvement over their corresponding original activation functions (i.e. , AVR and Linear). Therefore, the fractal and concatenated structure played an important role in activation functions.

To further evaluate the effects of excitatory and inhibitory neurons with different rations, we adopted the addition of the first excitatory neuron and the third inhibitory neuron, i.e. , ELU [Clevert et al., 2015] whose activation function is f(x) = f1(x) +αf7(x), for the baseline. Here, the adaptive penalized parameter α was set to different values. As shown in Table 5, we found that as the parameter α changes, ELU had significant performance differences in CIFAR-10 and CIFAR-100 datasets. While α = 0.11, ELU achieved the best performance. This was corresponding with the neurological guess in Section 3.2. In some extent, it showed the relevance between computer science and neurological evidences.

4.6 Comparison with other State-of-the-Art Activation Functions

We compared the excitatory and inhibitory activation (EIA) function with other state-of-the-art activation functions, i.e. , ReLU [Nair and Hinton, 2010], PReLU [He et al., 2015], CReLU [Shah et al., 2016] and PELU [Trottier et al., 2016]. VGG-Large-2 was increased the filter number as fair baseline for comparison with CReLU and EIA. From Table 6, we found that the EIA function achieved the best performance. Specifically, EIA not only took advantage of the phase information and modulus information of different fractal quadrants, but also utilized the complementarity of excitatory and inhibitory neurons.

4.7 ImageNet Challenge Dataset

To evaluate the performance of our proposed activation functions in large dataset benchmark, we performed the activation function of EIA on the 1000-class ImageNet dataset, which consists of about 1.3M training color images. For this task, we adopted AlexNet [Krizhevsky et al., 2012] with batch normalization for test. In Table 7, we found that our proposed activation function achieved the best performance (i.e.

, the highest testing accuracy) compared with the state-of-theart activation functions ReLU [Nair and Hinton, 2010] and CReLU [Shang et al., 2016] in the large ImageNet benchmark dataset. The experiments also demonstrated the effectiveness of the proposed EIA function.

5 Discussion

In this section, we discussed the relationship between our findings and neuroscience. For parallel and distributed biological neurons, the expected firing rate is simulating the activation function while there are signals at synapse incoming [Dayan and Abbott, 2001; Glorot et al., 2011]. The biological activation function consists of features: unilateral inhibition, sparse activation, and antisymmetric inhibitory or symmetric excitatory states. Inspired by biological knowledge, the proposed EIA function had relations with the above features. Fractal assumption was similar to unilateral inhibition, while the concatenated structure in parallel used sparse activation and inhibitory/excitatory states potentially. In some extent, the compositional activation functions provided understanding and guidance for future neurological findings.

6 Conclusion

In this paper, we proposed two types of neurons with different activation functions for artificial neural networks: excitatory and inhibitory neurons. With the help of different types of neurons, we designed the excitatory and inhibitory activation function which enabled the networks to be more sparse and alleviated the bias shift and vanishing gradients problems. Comprehensive experiments were not only in agreement with neuroscience findings, e.g. , the importance of excitatory and inhibitory neurons, but also demonstrated the proposed method had achieved state-of-the-art performance.

Acknowledgments

This work was supported by the National Natural Science Foundation of China under Grant 61375036, 61511130079, 61403380, 61602481, and the Strategic Priority Research Program of the Chinese Academy of Sciences under Grant XDB02060009. Zhaoxiang Zhang is the corresponding author of the paper.