Modeling Hebb Learning Rule for Unsupervised Learning

1 Introduction

A promising but challenging research in artificial intelligence has been mimicking the process the human brain represents information for decades [Arel et al., 2010]. Neural science findings [Kruger et al., 2013] have provided new ideas for designing information representation systems. This has motivated the development of hierarchical learning architectures and feature learning models.

Unsupervised feature learning aims to learn good representations from unlabeled input data. One of the interests in unsupervised learning is the stacked single-layer learning models for learning hierarchical representations [Hinton et al., 2006; Hinton and Salakhutdinov, 2006; Bengio et al., 2006; 2013]. Unsupervised learning had a catalytic effect in reviving interest in deep neural networks [LeCun et al., 2015]. Although it has little effect in many purely supervised applications, it still works when large amount of labeled samples are unavailable. Various single-layer feature learning models have been used and proposed since the layer-wise training by restricted Boltzmann machine (RBM) [Hinton et al., 2006] and auto-encoder (AE) [Bengio et al., 2006], including principal component analysis (PCA) [Valpola, 2014], sparse coding [He et al., 2013], denoising AE [Vincent et al., 2010], and sparse versions of AE and RBM [Lee et al., 2007; Ji et al., 2014; Ranzato et al., 2007; Gong et al., 2015].

∗Corresponding author

Good representations eliminate irrelevant variabilities of the input data, while preserving the information that is useful for the ultimate task [Ranzato et al., 2007]. For this purpose, an intuitive way is to construct the encoder-decoder model. AE, RBM, PCA and most unsupervised single-layer feature learning models follow this paradigm. Based on the encoderdecoder architecture, more useful features can be learned by adding constrains to extract more abstract concepts, including sparse constraints and sensitivity constraint [Rifai et al., 2011]. Another type of learning models only have a decoder, sparse coding for example. The above models can be taken as task based models which model the task and find the solution that can well handle the task. However, some tasks are amphibolous to define and model. The feature learning task for instance, there are many ways to define the abstraction, e.g., sparse representation, noise corruption [Vincent et al., 2010], and contractive constraint [Rifai et al., 2011].

In this paper, we propose a learning rule based model which attempts to model the learning process of neurons with less considerations of the task. The model consists of only the encoder. The learning behaviours of neurons have been researched for a long time for revealing the mechanism of human cognition. One of the most famous theory is the Hebb learning rule [Hebb, 1949] proposed by Donald Olding Hebb. By applying the Hebb rule in the study of artificial neural networks, we can obtain powerful models of neural computation that might be close to the function of structures found in neural systems of many diverse species [Kuriscak et al., 2015]. Recent works on Hebb learning are the series on "biologically plausible backprop" [Scellier and Bengio, 2016] by Bengio, et al. They linked Hebb and other biologically formulated rules to back-propagation algorithm and used them to train the neural network. Hebb rule gives the updating gradient of the connecting weights and it is unsupervised itself. We establish the model with the single layer network as many unsupervised learning models and attempt to model the updating gradient as the objective function. However, the architecture of artificial neural network is highly simplified from the neural network in brain. Therefore, when the learning rule is implemented, several properties have to be considered [Kuriscak et al., 2015] in order to adapt to and reinforce the simplified network. We call the model neuron learning machine (NLM) and find that NLM has many similar properties to other unsupervised feature learning models. From the analysis and experiments, NLM is capable of learning useful features and outperform the compared task based models. For learning features from 2-dimensional data, e.g., images and videos, we also extend the model into a convolutional version.

This paper is organized in five sections. In the next section, the proposed NLM is formulated. In Section 3, we explain how the NLM works and learns useful features. The experimental studies are provided in Section 4. In Section 5, we conclude this paper.

2 The Model

The structure of NLM follows the common structure of single-layer neural network, e.g., the encoder of the AE and RBM. The values of hidden units are computed by:

where h is the hidden vector and v is the visible vector. W and b denote the connecting weight matrix and bias vector respectively. s is the activation function and here we use the sigmoid function s(x) = 1/(1 + exp(-x)).

Hebb learning rule can be summarised by the most cited sentence in [Hebb, 1949]: "When an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A's efficiency, as one of the cells firing B, is increased." In the training of an artificial neural network, Hebb rule describes the updating gradient of each connecting weight Wij = Wij +∆Wij . This can be simplified by the product of the values of the units at the two ends of the connection:

where α is the learning rate. Hebb rule itself is an unsupervised learning rule which formulates the learning process of neurons. Therefore, we attempt to model the network based on this learning rule in order to learn in the neuronal way.

With the gradient, a model can be obtained by integrating over ∆W. However, since W is a matrix and h actually is a function of W and v, i.e., h = f(W, v), it is of great difficulty to derive the integral. As discussed above, the artificial neural networks are highly simplified models inspired from real neural networks. Therefore when the Hebb learning rule is applied to the simplified network, many considerations have to be considered, i.e., locality, cooperativity, weight boundedness, competition, long term stability, and weight decrease and increase [Kuriscak et al., 2015]. With these considerations and constraints, we can simplify the integration process. In real neural networks, there are many factors influencing the status of hidden units, including status of visible units, information from other nucleus, competition between hidden neurons and adjacent connections and limitations of the neuron itself. Then, a connection of the simplified network has less influence on h. Therefore, for convenience, we assume that the connecting weight has negligible influence on the connected hidden unit. Under this omitting assumption, h can be taken as a set of constants wrt W and the integral is apparent, i.e., max ∑ij ∫Wij hivj = ∑ij Wijhivj = vT Wh. Although this term is derived under the hypothesis, optimizing this term will increase the correlation between Wij and vjhi which is consistent with the Hebb rule. Specially, this term is also similar to the energy defined in RBM and Hopfield network.

However, without the probabilistic model in RBM and the recurrent architecture in Hopfield network, maximizing this term merely would not work because the simplified network cannot approximate this assumption and guarantee the stability of solutions without any constraints. Components in W will be +∞ or -∞ when vT Wh is maximized. In order to make the network approximate the assumption and guarantee the long term stability of W and h, a correlation based constraint is introduced. The assumption omits the effect of W on the hidden units h. Therefore, both positive and negative correlations between W and h should be minimized which can be represented by the cosine between the two vectors cos2 (h,W:j) where W:j is the j-th row of the matrix W. Meanwhile, the length of W and h should be limited in order to prevent the infinite increase or decrease of W. Then the correlation constraint can be formulated as the square of inner product between the two vectors ∑j 〈h,W:j〉2 = ∑j ‖W:j‖22 ‖h‖22 cos2 (h,W:j) = ‖Wh‖22 . With the simplified Hebb model and the correlation based constraint, we can obtain the objective function of NLM:

where λ is a user defined parameter that controls the importance of the two terms. Here we explain the assumption with neuron competition which is inspired from lateral inhibitory.

The network can be taken as undirected graphical model with connections representing the relationship between visible and hidden units as shown in Figure 1. Without competition, the hidden units are independent identically distributed. Because Hebb learning has the property of locality where the update of a weight depends only on the preand postsynaptic neurons. The max-clique of this graph is circled by the red ellipse in Figure 1 (a). When the competition is implemented, there are implicit connections between each pair of hidden units. Then the max-clique includes all the hidden units as shown in Figure 1 (b). Following the probability distribution defined in RBM, we give the energy of the max-clique with the connecting weight E(C) = ∑xi,xj∈C E(xi, xj) = - ∑xi,xj∈C xiWijxj where C denotes the clique and the energy of the clique means the sum of the energy of each connection. The potential of a clique is defined as Φ(C) = exp(-E(C)) = ∏xi,xj∈C Φ(xi, xj). With the potential, the joint probability density of the graph p(x1, x2, ...) = ∏C Φ(C)/ ∫x1,x2,... ∏C Φ(C). Therefore the distribution of the network in Figure 1 (a) is denoted by:

With the simplified architecture, when only Hebb learning is considered, the hidden neurons are independent to each other and all prefer to respond to the most frequent pattern. When the competition is complemented, the potential of a max-clique with vj is denoted by Φ(Cj) = ∏i Φ(vj, hi) ∏k,l Φ(hk, hl). The joint probability of this network is then computed by:

where n is the number of visible units. In Eq. (4) and (5), the potential Φ(vj, hi) depends on the status of units as well as Wij. The potential Φ(hk, hl) is then depends on the neuron competition. In neuron competition, the neurons inhibit the firing of other neurons. Therefore, the latent connecting weights between them are usually negative values. Then the summed energy of hidden units ∑kl E(hk, hl) = ∑kl -hkWklhl will be a positive large value. The status of the hidden units follows the conditional probability p(h|v) = p(v, h)/ ∫h p(v, h) = ∏j Φ(Cj)/ ∫h ∏j Φ(Cj) and therefore with competition, it depends largely on the competition dominated potential while less on the network parameter W. Consequently, we make the assumption in order to model the Hebb rule conveniently.

Since the proposed model has only an encoder, it is convenient to derive a convolutional one in order to learn useful features from images. In convolutional neural networks (CNN), the feature maps are obtained by several trainable convolution kernels:

where Hi denotes the i-th feature map and Wi denotes the i-th convolution kernel. The convolutional NLM (CNLM) can be obtained by summing over all the pixels in feature maps. Then the first term can be denoted by ∑(x,y) ∑i (I ∗ Wi)(x, y)Hi(x, y) = ∑i tr(I ∗ WiHT i ), where (x, y) denotes the pixel position in H and tr() is the trace of a matrix. For the second term, the convolutional version is also easy to derive, i.e., ∑(x,y) ∑(i,j) 〈W(i, j),H(x, y)〉2 , where (i, j)

denotes the position in convolution kernels. By combining the two terms, the CNLM is formulated:

NLM and CNLM can be optimized by the widely used stochastic gradient descent algorithm.

3 Analysis

In this section, we explain how the NLM learns useful features from input data. As introduced above and described in [Vincent et al., 2010], one natural criterion that we may expect any good representation to meet, at least to some degree, is to retain a significant amount of information about the input. As introduced above, the encoer-decoder and decoder based models can achieve this criterion because of the reconstruction of input data, i.e., max p(v|h). NLM has only the encoder, but the neuron inspired learning process can also achieve this criterion. In NLM, maximizing Eq. (3) amounts to increasing the cosine correlation coefficient between v and W h:

Since v is the observed data, maximizing Eq. (3) means to increasing the numerator and decreasing the denominator and therefore increasing the correlation coefficient. Then the conditional probability p(v|Wh) is enlarged. With a distribution of visible data q(v), Eq(v) [log p(v|Wh)] can be increased. p(v|Wh) is a parametric distribution parameterized by W, therefore it can be written as p(v|h; W). With the deterministic mapping from v to h, Eq(v) [log p(v|Wh)] can be written as Eq(v,h) [log p(v|h)]. For any distribution p(v|h) we have [Vincent et al., 2010]:

Eq(v,h) [- log q(v|h)] denotes the conditional entropy H(v|h) between v and h, and the mutual information can be computed by I(v; h) = H(v) - H(v|h) with H(v) being a constant. Therefore, optimizing the objective function in NLM amounts to increasing the lower bound on the mutual information between v and h. This is similar to AE [Vincent et al., 2010]. Consequently, the representation h learned by NLM can retain information of input data v.

Another criterion of a good representation is to eliminate irrelevant variabilities of the input data, i.e., abstraction and invariance [Bengio et al., 2013]. Many models achieve this criterion by introducing regularization terms, e.g., sparsity induced term [Lee et al., 2007; Ji et al., 2014; Ranzato et al., 2007; Gong et al., 2015] and sensitivity penalization [Rifai et al., 2011]. By introducing noise, the model can be trained to increase the robustness to local variation [Vincent et al., 2010]. This can also learn useful features. In NLM, the abstraction can be achieved by eliminating irrelevant components in v. For each visible unit vj, the squared inner product between h and W:j is minimized which breaks the relationship between h and vjW:j as well. Then the conditional probability p(h|vjW:j) will be minimized. Since h = f(W, v), the conditional probability can be written as p(h|vj;W:j). The constraint term is the sum over all the vj , and then minimizing the constraint term amounts to minimizing the summed conditional probability ∑j p(h|vj ; W:j ). However, maximizing the first term will increase the correlation between vT W and h which increases the conditional probability p(h|v;W). As a consequence, by maximizing the objective function in NLM, p(h|v;W) is increased while the summed probability over all the components in v, i.e., ∑j p(h|vj;W:j) is reduced. This means that h can respond to v while h cannot respond to most of the components in v. In other words, in NLM, h responds to most relevant components which omitting the relatively irrelevant components in v. Therefore, the proposed NLM has the ability of abstraction.

From another perspective, in NLM, the first term is similar to the energy defined by RBM:

RBM is a generative model and the energy based joint distribution is considered:

where Z is a normalization constant. RBM is trained by maximizing the probability that the network assigns to the observed data, i.e., p(v) = ∑h p(v, h). The probability that the network assigns to a training data can be raised by adjusting the weights and biases to lower the energy of that data and to raise the energy of other data, especially those that have low energies and therefore make a big contribution to the partition function [Hinton, 2010]. Similarly, NLM can be taken as an energy based model with a correlation based constraint. Considering negative version of the first term-vT Wh as the energy of the network, the constraint can be used to restrain the decrease of the summed energy of all the data. Since h responds to v according to W, by minimizing the squared inner product between h and W:j , the summed energy ∑d -dT Wh over all the possible data d in the data space is raised to approach to 0 because most components in Wh is restrained to 0. However, since the first term is the energy of the observed data, such energy is relatively lower than that of other data. From the perspective of network energy, the proposed NLM assigns high probability to the observed data which is similar to RBM.

4 Experimental Study

4.1 Experiments on the Learning Process

First we implement the experiments on the gradient based learning process and verify the claims we make in the analysis of the model. The widely used MNIST digit dataset is used here and we randomly select 10000 images from the training set in order to speed up the experiments. We claim that optimizing the objective function in NLM amounts to increasing the lower bound on the mutual information between v and h. This can be verified by the correlation coefficient between v and Wh in Eq. (8). Therefore we plot the value of cos(v,Wh) and ∑j cos2 (vjW:j, h) = ∑j cos2 (W:j, h) in each iteration as shown in Figure 2 (a). The blue dashed line represents ∑j cos2 (W:j , h) and red solid line represents cos(v,Wh). During the learning process, the correlation between v and Wh increases. This demonstrates that NLM achieves similar function to AE. Then on the squared correlation coefficient between W and h, with the increase of iterations, it gradually converges to 0. This means that the correlation constraint makes the network approximate the assumption and reduce the summed conditional probability ∑j p(h|vj;W:j).

We also claim that in NLM, h responds to most relevant components which omitting the relatively irrelevant components in v. Then we select two images from the MNIST dataset and CIFAR-10 dataset respectively, and plot cos2 (h,W:j) for each j as the response to each vj during the iterations. Figure 3 shows such plots of the initialization, 10th iteration (N=10), 50th iteration (N=50) and the last iteration (N=100), respectively. Since v is 2-dimensional data, the plots are exhibited in the 3-dimensional form with the heave representing the response to each pixel in v. The sharp peak demonstrates that h responds to a few components in input data and most components cannot significantly influence the response of h and W. For the image in CIFAR-10 dataset, we also compare the response plots with different λ.

From the network energy perspective, NLM has the similar properties to RBM. We plot the energy -vT Wh in each iteration of the observed data set and other data set as dotted line and dashed line respectively show in Figure 2 (b). The other data set is a random set with the same size to

5

20

15

10

0

10

5

0

20

15

10

5

0

30

25

20

15

10

5

0

Figure 3: 3 dimension plots of cos2 (h,W:j) on each pixel of the input image. Two images from the MNIST dataset and the CIFAR10 dataset respectively are exhibited. With the increase of iterations, i.e., N=0, 10, 50, and 100, the irrelevant pixels are restrained. With different λ, the response intensity of each pixel is different.

the observed data set. With the learning process, the energy of observed data decreases greatly while that of the other data does not significantly change. To further narrow the distance between NLM and RBM, we set a simplified p(v) = exp(vT Wh)/(exp(vT Wh) + exp(dT Wh)), where d denotes the other data. p(v) in each iteration is plotted into red line as shown in Figure 2 (b). p(v) increases with the learning process which demonstrates that NLM can model the observed data well.

4.2 Experiments on Single-layer Network

Above experiments show the behaviours of NLM during learning process and verify the claims we make in Analysis. Then we test NLM on single-layer network and the features learned by NLM are evaluated. The experiments are implemented on the MNIST digit dataset and image patches. The image patches with the size of 12 × 12 are randomly extracted from two remote sensing images respectively. Some digit images and the two remote sensing images are shown in Figure 4. The first image is a panchromatic image acquired by Landsat 7 satellite which shows the city of Wuhan in China. The size of the image is 4561×4505 pixels and we randomly extract 100000 image patches as the Wuhan dataset. Another image is a synthetic aperture radar (SAR) image which shows the Yellow River Estuary in China. The size of this image is 3233 × 5193 pixels and there are 100000 image patches in the Yellow River dataset. Here the remote sensing images are used because of the significance and difficulty of interpreting remote sensing images. Some randomly selected bases learned by NLM on the three datasets are shown in Figure 4. AE, RBM, and the recently proposed sparsity induced version of them, i.e., multiobjective sparse feature learning model (MO-SFL) [Gong et al., 2015] and sparse response RBM (SR-RBM) [Ji et al., 2014] are compared. SR-RBM is based on RBM and use the l1-norm of the activation probability of hidden units to achieve a small code rate. A user defined parameter is used to control the importance of the two terms. MO-SFL considers the difficulty of determining the compromise between reconstruction error and sparsity. MO-SFL is more adaptive than SR-RBM.

First on the MNIST dataset, without constraints, AE and RBM achieve unstructured weight patterns. Sparsity is a model of simple cells in visual cortex and it has been a key element of unsupervised learning algorithms. With the sparsity constraint, the uncertainty of representations is re-

15

25

35

30

40

duced which increases the robustness to irrelevant variabilities. Therefore, the bases learned by them can capture the local structure of the images as shown in the figures. In NLM, the constraint term leads the representation to responding to most relevant components. Therefore, the local structure can also be captured by the weights. Then on the remote sensing image patches, it is more difficult to deal with remote sensing images than nature images because such images are acquired from satellite and many interferences may influence the quality of the images. As a consequence, AE, RBM and their derivatives may be misled by the inferences. Some bases learned by NLM are Gabor-like edge detectors in different positions and orientations. The results are consistent with many works on nature image patches.

In practice, a good representation is one that will eventually be useful for addressing tasks of interest. Therefore, we test the features learned by the task of image classification. The experiments are implemented on the MNIST dataset and in order to speed up the learning process, we randomly sample 10000 images from the training set to train NLM as well as other models. For the models, we set the number of hidden units to 196. In NLM, λ = 1 and the learning rate in stochastic gradient descent algorithm is set to 0.01. Those models keep the same number of iterations. The features learned by those models are fed into a linear classifier and we randomly sample 1000, 5000, and 10000 images to train the classifier. The test error rates (%) on the test set are listed in Table 1. The best result is highlighted in bold. From the classification test, sparsity induced models can outperform the base models. NLM always achieves the best result.

In NLM, the user defined parameter λ may influence the performance. Therefore, we set λ = 0.03, 0.04, ..., 1.9 and plot the error rate in Figure 2 (c). The result of MO-SFL is marked by a red line for comparison. We can see that too small or too large values of the λ can reduce the performance of NLM. However, NLM can outperform MO-SFL with a large range of λ values, from 0.5 to 1.9. Meanwhile, relatively good performance can also be achieved with many λ values, from 0.8 to 1.1. Consequently, NLM is not too sensitive to λ.

Another parameter that may influence the performance of learning models is the number of hidden units. In the above experiments, 196 is the empirical value. We set the number of hidden units to 300, 400, and 500 for these learning models and plot the error rate as shown in Figure 2 (d). With the sparse constraint, AE and RBM achieves much better performance when there are more hidden units representing the input data. NLM with 196 hidden units outperforms the other models with 300 hidden units which demonstrates the effectiveness for using hidden units to represent input data.

4.3 Experiments on Deep Networks

Above experiments evaluate NLM on single-layer network. Then we apply NLM to deep neural networks and learn hierarchical representations. First two NLMs are stacked with the hidden units being 500 and 300 respectively. We set λ = 1 and the learning rate to be 0.01. The compared models, i.e., AE, RBM, SR-RBM, and MO-SFL, are also stacked with the same architecture. The models are trained by the 60000 training images in the MNIST dataset. The learned hierarchical features are fed into a linear classifier and we use randomly sampled 10000 images and 60000 images to train the classifier. The test error rates are listed in Table 2. With the hierarchical representation, NLM can outperform the compared models. However, the performance improvement is much less than single-layer network because of the limited architecture for dealing with images. For efficiently learning features from images, we use the architecture of CNN and learn the features by CNLM.

First on the MNIST dataset, the input of the CNLM is 32 × 32 images which are obtained by evenly zero padding original images. There are 10 5 × 5 filters in the first layer and followed by a 2 × 2 max-pooling layer. The feature maps in the 3rd layer are obtained by 50 10 × 5 × 5 filters and the 4th layer has 50 down-sampled feature maps by maxpooling. Finally, 200 50 × 5 × 5 filters are implemented and a feature vector is obtained. Then the features are classified by SVM. We compare CNLM with the convolutional version of AE [Masci et al., 2011] and RBM [Norouzi et al., 2009]. The test error rates are listed in Table 2.

Then we implement CNLM on a more difficult dataset, i.e., the CIFAR-10 dataset. This dataset consists of 50000 training and 10000 test color images with the size of 32×32. The hierarchical architecture has 5 layers with 3 convolution layers and 2 max-pooling layers, i.e., 3-100-100-150-150-200. The size of filters in convolution layer is 5 × 5 and the pooling size is 2×2. The learned features are classified by SVM. The convolutional versions of AE and RBM are compared. The test errors are listed in Table 2. Experiments on deep neural networks demonstrate the effectiveness of NLM for learning hierarchical features.

5 Conclusions

In this paper, a neuron learning machine (NLM) was proposed based on Hebb learning. Hebb learning rule describes how synaptic weight changes and can be formulated by the gradient of a connecting weight in neural network models. To model the network with Hebb learning, we integrate over the gradient by assuming the negligible effect of weights on representations. This omitting assumption simplifies the integral process greatly. In order to approximate this assumption and obtain a stable solution, a correlation constraint is added. We find that NLM has the properties of retaining information and abstraction which are essential for learning useful representations. Analysis and experiments demonstrate the effectiveness of the learned features by the proposed model. The future work will include but not limited to exploring the learning behaviours of neurons for efficient unsupervised as well as supervised neural learning models on various network architectures.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (Grant no. 61422209, 61672409), and the National Program for Support of Top-notch Young Professionals of China.