Theory of Self-Organizing Nerve Nets with Special Reference to Association and Concept Formation

1 . INTRODUCTION

The brain needs a long period of evolution to get the present information processing manner. Although artificial intelligence is recently investigated rather independently of the behav­ ior of nerve nets, it is plausible that artifi­ cial and natural intelligence have some common logic of realizing intelligence at a higher abstract level. They have something to suggest to each other. Neural systems seem, especially, to yield good models of parallel and distributed information processing with self-organizing £apabilities of adapting themselves to the en­ vironmental information structure.

The present paper proposes a unified mathemati­ cal treatment of neural self-organization applicable to wide classes of learning with and without teacher, by summarizing the author's recent researches [1 - 5]. We also present a model which elucidates the mechanism of unsuper­ vised and automatic formation of various routines for processing the signals in the environment [cf. 4, 6, 7], and analyze the model by the proposed method. Recent developments of neuro­ physiology have revealed the importance of such self-organization. We also touch upon the models of topographic organization of nerve fields [3, 8] as· well as of associative memories [9. 10].

2. A GENERAL THEORY OF SYN.I\PTIC MODIFICATION

We consider a simple mathematical model of a neuron, which receives a vector input signal ~ = {x ·...· x ) from its environment s. and emits on! output ~ignal z. These signals in general take analog values between 0 and 1, representing the normalized pulse emission rates {they may take on 0 and 1). Let us denote by~= {w1 ·...· wn) the synaptic efficiencies or weights of the inputs. The input-output relation is given by

where h is a threshold, f is a monotonically increasing function transforming the average Membrane potential u =~·~into the output pulse emission rate z {Fig.l).

Tht=! neuron self-organizes and adapts to its environment S, by modifying w based on an input time sequence x{t) from S. Tn some cases, the so called teacher signal y{t) is associated with x{t). Hence we regardS as an er9odic informa­ tion source producing {x{t). y{t)). where teacher signal y{t) is absent in the case of unsupervised learning. We consider a simple environment S represented by a probability {density) p{~. y). It chooses a pair {x. y} of signals with the probability p{x, y): each time independently, and outputs it for-a fixed time duration, and then choose another, repeating this process.

We propose the following general rule of synapse modification, ~ being changed based on ~{t) as

where T is a large time constant, "·" denotes the time derivative d/dt, c is a constant, and r is the reinforcement or learning signal which the neuron produces depending on w, x andy (when it exists) at that time, r(t) = r[w(t), x(t), y(t)]. This is a generalization of HebbTan type learning, ·-1hich is obtained by putting r = z. Perceptron type learning is obtained by r = y - f(~·~- h). Most of the neural learning rules proposed so far are obtained by chasing appro­ priate signals r.

By replacing r~ in (2) by its average, we obtain

where < >s denotes the average with ~espect to p(x, y) of S, where w is considered as constant. This is not a random equation. Since S is ergodic, we can expect that the real behavior of w(t) of (2) is closely approximated by that of \3). This really is proved by the method of stochastic approximation or by the theorem of Geman [11]. Hence, we consider the average learning equation (3) as the fundamental of neural self-organization.

When r depends on ~and ~through ~·~only, r = r(~·~· y), (3) is rewritten in the form

We call L(~) the potential of learning under S. The synaptic weight ~converges to one of the minimum of L. In most supervised cases, L has only one minimum, ~converging to it. However, in most unsupervised cases, L has a number of minima corresponding to various aspects of S, and w converges to one of them. However, when a pool of neurons receive common inputs from S, every minimum of L is occupied by some neurons in the pool, so that the pool of neurons adapts as a whole to the environment S.

3. FORMATION OF CATEGORY DETECTING CELLS

Information processin~ routines are formed in the brain to be compatible with the environment by learning. This kind of learning without teacher underlies commonly in the problem of concept formation, formation of feature detectors, etc. Recent developments of neuroohysiology have revealed that the feature detecting cells are really formed depending on the visual ex­ periences of an animal. We propose a simple model, and analyze its capabilities.

The model consists of a set of neurons receiv­ ing a common input signal x from the environment S = fp(~)l. They also receive an inhibitory input of constant intensity x0 (Fig.2). Let us take one neuron, and let w be the modifiable synaptic weights of the neuron. Let -w 0 be the modifiable synaptic \'Ieight for the inh1bitory input x0.

We assume the following learning rule that the learning signal r is equal to 1 when the neuron is excited (z > 0) and 0 other~ise (z = 0), i.e.,

whereZ(u) is 1 when u > 0 and 0 otherwise, and we pnt h = 0. The average learning equation is

The equations are in general multi-stable, hav­ ing a number of equilibria.

Let us consider a subset A of the signals in S. A nerve cell is said to be a detector or a repre­ sentative cell of A, when it is excited by any signal in A but not excited by any others in S. We say that a subset A forms a category under S, \"/hen a detector of A is obtained as a stable equilibrium by self-organization. In other words, a subset A is a category, when a set of weights ~A,w 0 A satisfying

for all ~e A but not for any ~E S-A, is obtained as stable equilibrium of (7) and (8).

Many categories are formed by learning under S. When the neuron pool includes many neurons of different initial weights, theY. differentiate automatically to be detectors (representatives or processors) of these categories. Mutual interaction among the neurons of lateral inhibi­ tion type prevents a large number of neurons becoming the detector of one and the same cate­ gory. The problem is to know which subsets become categories under S. We give a n.a.s.cA for a subset A to be cateqory under S. Let x be the average of signals in A. xA = J'p(x)xdx. Let ~A(~) = ~A·~ be the inner product, and i ~ c'xotc.

Theorem. A necessary and sufficient condition that A is a category under S is The theorem shows that a cluster of signals of an adequate diameter forms a category, where the diameter is specified by the parameter A. In other words, A specifies the resolution of a category, where A can be controlled by changing xo. This yields a primitive mechanism of forma­ tlon of information processing routines by 1 earning.

4. TOPOGRAPHIC ORGANIZATION

Topographic organization is found in many parts of nerve fields in the brain. For example, the continuous projection from the retina to the striate cortex is known as the retinotopy. It is known that, when part of the retina (or striate cortex, or both) is removed, the regen­ eration of connections takes place such that the topological ·map is completed between the remain­ ing parts. This suggests that topographic organization is formed based, at least partly, on the self-organizing ability of nerve cells.

Let us consider a simple model consisting of a presynaptic nerve field X and postsynaptic nerve field Y connected by modifiable synapses. Stimuli are supplied from the environmental in­ formation source S to the presynaptic field X, exciting neurons of X, and then those of Y. The self-organization takes place based on these excitations. In this case, S brings the topo­ logical information of X, in such a way that two nearby neurons of X are frequently stimulated at the same time. When the neurons in Y have fixed mutual connections of lateral-inhibition type and when they have additional modifiable inhibitory synapses, we can prove that continu­ ous connections from X to Y (topographic organization) is formed by Hebbian type unsuper­ vised learning. Moreover, we can prove that some microstructures are also formed in some cases, as is shown in the columnar structures in the cerebrum. The fundamental equations of the model are much complicated, because we should solve the field equations of neural excitation together with the average learning equations. We can solve them.

5. PRIMITIVE MODEL OF ASSOCIATION

Let us consider a pool of neurons, which receive a common input signal ~and output a vector signal z (the i-th neuron emitting the i­ the component zi). We consider such an envi­ ronment S that consists of a finite number of signal pairs (~, lj), where the i-th component Y·· of lj acts as tne teacher signal to the i-th n~~ron. The 'problem is to modify the synaptic weights ~i of the i-th neuron (or synaptic weight matrix W = (~i)) such that, receiving an input ~x·, the neuron pool outputs the associated signal ·. In this case, one many say that the net reca ls ~i from the associated ~i· and that the associateo pairs (~, lj)'s are memorized in the synaptic weights W 1n a distributed and superimposed manner [10].

The correlation learning [10] is obtained, if each neuron self-organizes with r = y. In this case, W converges toW= E ljT~, T denoting the transposition. This learning works well only when ~ 's are mutually orthogonal. The synaptic welghts converges to W = E tJ·~* by orthogonal learning [2] with r = y - W·X, where {~j*} is the dual orthogonal system of{~} (i.e., ~*·~i = Oji), and associative recall can be done ~ven when x·'s are not orthogonal. We can also treat the~ynamic recall of signal sequences ~ 1 · ~ 2 .... , ~t [9].

~EFERENCES

1. S. Amari "A Method of Statistical Neuro­ dynamics". Kybernetik, 14 (1974) 201-215
2. S. Amari "Neural Theory of Association and Concept-formation". Biol. Cybernetics, 26 (1977) 175-185
3. S. Amari "Topographic Organization of Nerve Fields". to appear
4. S. Amari and A. Takeuchi "r1athematical Theory of Formation of Category Detecting Nerve Cells". Biol. Cybernetics, 29 (1978) 127-136.
5. A. Amari and M.A. Arbib "Competition and Cooperation in Neural Nets". In Systems Neuroscience (J. Netzler ed.), Academic Press, (1977) 119-165
6. C. von der Malsburg "Self-organization of Orientation Sensitive Cells in the Striate Cortex". Kybernetik, 14, (1973) 85-100
7. K. Fukushima "Cognitron: A Self-organizing Multilayered Neural Network". Biol. Cybernetics, 20 (1975) 121-136 ------
8. A. Takeuchi and S. Amari; to appear [9] S. Amari "Learning Patterns and Pattern Sequences by Self-organizing Nets of Thresh­ old Elements". IEEE Trans., C-21 (1972) ll97-1206
9. T. Kohonen Associative Memory. Springer­ Verlag, 1977
10. [ll] S. Geman "Some Averaging and Stability Results for Random Differential Equations". SIAM J. Math. 36 (1979) 86-105