Self-Organization of a Neural Network which Gives Position-Invariant Response

1. INTRODUCTION

In this paper, I propose a new algorithm for self-organizing a neural network which has an ability to recognize stimulus patterns based on the geometrical similarity of their shapes regardless of their positions and slight

distortions of their shapes.

This network is given a nickname "neocognitron", because it is a further extension of the "cognitron", which is a self-organizing multi­ layered neural network proposed by the author before [1]. Incidentally, the conventional cognitron .also had an ability to recognize patterns, but its response was dependent upon the position of the stimulus pattern. That is, the same patterns which were presented at different positions were taken as different patterns by the conventional cognitron.

The neocognitron has also a multilayered structure. It has an ability of unsupervised learning: We don't need any "teacher" during the process of self-organization, and it is only needed to present a set of stimulus patterns repeatedly to the input layer. After completion of self-organization, the network acquires a structure similar to the hierarchy model of the visual nervous system proposed by Hubel and Wiesel [2], [3]. The response of the cells of the last layer of the network is dependent only upon the shape of the stimulus pattern, and is not affected by the position of pattern presen­ tation. That is, the network has an ability to recognize patterns without affected by the position of the patterns.

2. STURUCTURE OF THE NETWORK

The network which is proposed here consists of a photoreceptor layer followed by a cascade­ connection of a number of madular structures, each of which is composed of two layers of cells connected in a cascade. The first layer of each module consists of "S-cells", which correspond to simple cells or lower order hypercomplex cells according to the classification of Hubel and Wiwsel. The second layer of the module consists of "C-cells", which correspond to complex cells or higher order hypercomplex

cells.

The cells in each layer are divided into many subgroups according to the optimum stimulus features of their receptive fields. Since the cells in each subgroup are set in a two dimen­ sional array, we call the subgroup as a "cell­ plane". It is assumed that all the cells in a single cell-plane have the same spatial distri­ bution of the input synaptic connections, and only the position of the presynaptic cells are shifted in parallel from cell to cell. Hence, all the cells in a single cell-plane have receptive fields of the same functions, but at different positions.

Let IA.uCA1 , n) be the output of an S-cell in the ~~-th cell-plane in the l-th module, and let u,1 cA1 .n) be the output of a C-cell in that module, where n is two-dimensional co-ordinates representing the position of these cell's receptive fields in the input layer U0 · Besides these excitatory cells, we have inhibitory cells VstC111) and vaCnJ in the module.

The output of these cells are represented as follows:

Where, a.tcA1., . "'. ~) and .t,clr,J represent the strength (efficiency) of the modifiable excitatory and inhibitory synapses, respectively. Parameter c1., CIIJ and d1C11J represent the strength of the unmodifiable excitatory synapses. Positive parameter r1 controls the intensity of the inhibitory input, and the large value of ~ gives higher selectivity of the cell's response.

Let U0Cn) be the output of a cell of the input layer Uo. In the above equations, it is assumed that Uc1.,C!$1. ,,n) stands for u 0<n> in case of (=1. Incidentally, K~1 =1 for l=l.

The inhibitory cell Vc1. ,<nl has an r.m. s . -type input-to-output characteristic as i ndi cated by eq. (3). The employment of r.m.s . -type cell s is useful for endowing the network with an abil i ty to make reasonable evaluation of the s imilarity between the stimulus patterns. Its usefulness was analytically proved for a cogni tron (4] , and the same discussion can also be appl i ed to our new network.

3. THE SELF-ORGANIZATION OF THE NETWORK

One of the fundamental hypotheses employed in this network is the assumption that all the S-cells in a single cell-plane have afferent synaptic connections of identical spatial distribution, and that only the position of the presynaptic cells shift in parallel in accordance with the S-cell's position . Therefore, in eq. (1), the modifiable synaptic connection ct/1. 1 ., . ... , A 1 J is. assumed to be i ndepen­ dent of the position n of the presynaptic cell Ust<,,.nJ.

During the process of self-organization, several "representative" S-cells are selected every t ime when a stimulus pattern is presented . Here , we assume a certain mechanism which prohibit the selection of more than one representative from a single S-cell-plane. The detailed procedure for selecting the representatives are given later on .

Suppose a representative is selected from an S-cell-plane . The afferent synapses to the representative S-cells are reinforced wi th the same manner as in the case of r .m.s . -type cognitron (4), and the reinforcement of the synapses of all the other S-cells in that cel l­ plane follows those of their representative . These relations can be quantitatively expressed as follows.

Let cell u.uC~ , ,;J be selected as a representa­ tive. The modifiable synapses a1 cA 1 _,,..,,i,J and t 1 C~J , which converge to the cells of the ~1 -th cell-plane, are reinforced by the amount shown below:

where ~~ is a positive constant speci fying the speed of reinforcement.

The cells in the S-cell-plane from whi ch no representative is selected, however, do not have their input synapses reinforced at all.

The representatives are chosen by the fol l owing procedures. At first, we watch a group of S-cells in a layer whose receptive fields are situated within a certain small area in the input layer. Each group contains cells from all the S-cell-planes. We call the group as an "S-column". There are a lot of such S-columns in a single layer. From each S-column, the cell which has yielded the largest output is chosen as a candidate for the representatives. In case only one candidate appears from an S-cell-plane, the candidate is unconditionally selected as the representative from that S-cell-plane. If more than two candidates appear from an S-cell-plane, however, only the one which has yielded the largest output among them is selected as the representative from that S-cell-plane. If no candidate appears from an S-cell-plane, no representative is selected from that S-cell­ plane.

As it is seen from these discussions, if we consider that a single excitatory cell in the conventional cognitron [1] coresponds to a single S-c.ell-plane in the neocognitron, the procedures of reinforcement in the both systems are analogous to each other.

4. COMPUTER SIMULATION

The neural network proposed here has been simulated on a digital computer. In the computer simulation, we consider a seven layered network: U0 +U,,+Uct·U.sJ+Uc~+Un+UCJ, That is, the network has three stages of modular structures preceded by an input layer. The number of cell-planes in each layer is 24 for all the layers except Uo. The numbers of cells in these seven layers are 16xl6, 16xl6x24, lOxlOx24, 8x8x24, 6x6x24, .2x2x24, and 24 from the front. In the last layer UcJ, each cell­ plane contains only one cell, and that cell's receptive field covers the whole area of the input layer Uo.

We have presented five stimulus patterns, which are shown in the leftmost column in Fig. 1, repeatedly to the input layer Uo of the network. The positions of presentation of these stimulus patterns have been randomely shifted at every presentation.

After a certain number of pattern presentations, each stimulus pattern has become to elicit an output only from one Ucrcell, and conversely, this UcJ-cell has become selectively responsive only to that stimulus pattern. That is, none of the Ucrcells responds to more than one stimulus pattern. It has been confirmed that the response of layer UCJ is not affected by the position of a stimulus pattern at all. Neither is it affected by the slight change of the shape or the size of the stimulus pattern.

Fig. 1 shows some examples of the stimulus patterns which the neocognitron has correctly recognized: All the stimulus patterns in each row of Fig. 1 have elicited the same response to Uc 3 -cells. That is, the response of layer Uc3 is affected neither by shift of position like (a)-(c), nor by distortion of shape or size like (d)-(f), nor by some insufficiency of the pattern or some noise like (g).