Beyond PDP: The Frequency Modulation Neural Network Architecture

1 Introduction

One weakness of the traditional neural network architecture is commonly known as the binding problem. Another weak­ ness are difficulties in predicting and recognizing sequential patterns such as word sequences. Due to these problems in the traditional neural network models, application of the model to any serious natural language processing has not been conducted. In the Frequency Modulation Neural Net­ work (FMNN) 1 , one solution to the binding problem is at­ tained through propagation of the activation source informa­ tion through the frequency modulation of pulses generated by certain groups of neurons. Combined with local modular circuits which enable sequential prediction and activation, our model attains structured marker passing from the neu­ ral level. First, we briefly summarize the marker-passing and constraint propagation schemes which we assume as our basis of natural language and inferential processing and describe what needs to be attained by the neural network. Then we describe some features of the FMNN architecture relevant to such tasks.

· Also with the Laboratory for Computational Linguistics, Carnegie Mellon University.

t Also with NEC Corporation.

1 Also see [Kitano and Tomabechi, ms.] for mathematical details and the descriptions of some of the circuits not included in this paper.

2 Marker-Passing and Constraint Propagation

2.1 Marker-Passing Models

The spreading-activation marker-passing based models of cognitive processing reflect the highly interactive and par­ allel nature of human cognitive activities 2 · One of the ma­ jor differences between the marker-passing based models and the neural network models is that the marker-passing scheme allows some tokens (typically the sources of acti­ vation) to be passed with the activation propagations. The Direct Memory Access (DMA) model [Riesbeck and Mar­ tin, 1985] [Tomabechi, 1987] is a marker-passing scheme with case-based inference capability. In the DMA model, the spreading-activation of activation source tokens enables the models to instantiate generalized memory structures to capture the specific meaning of the input utterance. The tokenized activation source propagation can be seen as one way of solving the binding problem. Recently, researchers at Carnegie Mellon have adopted the DMA paradigm for var­ ious natural language and inference tasks. [Tomabechi and Tomita, 1988a] is a DMA natura} language interface and in­ ferences for development of knowledge-based systems and in [Tomabechi and Tomita, 1988b], DMA based contextual inferencing was integrated into a unification-based (LFG, [Kaplan and Bresnan, 1982]) phoneme-parser ([Saito and Tomita, 1988]) as a part of real-time speaker-independent speech-to-speech translation system ([Tomabechi et. al., 1989a]). Independently, a basic DMA algorithm has been implemented on VLSI chips [Kitano, 19881. Also, the first DMA based speech-to-speech translation has been pub­ licly demonstrated at Carnegie Mellon 3 {[Tomabechi et. al. , 1989b], [Kitano et. al., 1989b]).

2.2 Constraint Propagation

The Massively-parallel Constraint Propagation (MCP, [Tomabechi and Tomita, msJ) scheme extends the DMA phi­ losophy and proposes the models to propagate constraints as well as other information such as the tokens for the sources of activations. The Head-driven Massively-parallel Con­ straint Propagation (HMCP) [Tomabechi and Levin, ms.1 proposes a scheme of propagating head-feature constraints to increase DMA's capacity for handling syntactic phenomena such as word order, agreement, case marking, inflectional and expletive morphology, control, and unbounded depen­ dencies. In the MCP paradigm, words (or some smaller linguistic units) in the input string trigger the propagation of constraint information through the network. The propagated information includes the source of the activation, syntactic head-features, etc .. Concepts that represent heads of phrases contain bundles of syntactic features which constrain their complements. When concept activations of complements collide with activations of heads, the syntactic features of the complement and head are unified with each other.

2 Such as [Quillian, 1969], [Charniak, 1983], [Hirst, 1984], [Riesbeck and Martin, 1985], [Charniak, 1986], [Norvig, 1987], [Tomabechi, 1987], and [Hendler, 1988].

3 0ther efforts include the application of DMA to phoneme­ based parsing [Tomabechi et. al., 1988], discourse-based process­ ing [Kitano et. al., ms.], cost-based ambiguity resolution [Kitano et. al., 1989a] and concurrent parsing and generation [Kitano, 1989].

Below are the node entries taken from [Tomabechi and Levin, ms.1 representing4 the lexical concepts for the verbs give and try:

(lex-node *GIVE (is-a (*ACTION)) (phonology </g/ /i/ /v/ >) (s yn-head-feature ((MAJ V) (VFORM FIN) (AUX MI NUS))) (give r (*PERSON 1)) (rece i ve r (*PERSON 2)) (given (*OBJECT 3)) (subcat < (NP[NOM] 1), (NP[ACC] 2), (NP[ACC] 3 ) > )) (lex-node *TRY (i s -a (*ACTION)) (phonology </t/ /r/ /a/ /i/>) (syn-he ad- feature ((MAJ V) (VFORM FIN) (AUX MINUS))) (trier (*PERSON 1)) (circumstance (*ACTION 2)) (s ubc a t <(NP[NOM] 1), ((((MAJ V) (VFORM INF) ) subcat< (NP 1)>) 2) >))

The list (NP[NOM] 1) in the subcat feature is a short-hand for (((MAJ N) (CASE NOM)) 1).

The list in subcat represents the postulation for the con­ straints that are to be satisfied by the nodes that will fill the conceptual roles (that are co-numbered). The contents of the syn-head-feature are the head-features that are propagated with the specific node activations along with other informa­ tion. In the entry for try we can see that subcat specifies that it subcategorizes for an unsaturated verb phrase as an in­ stance to fill the circumstance role (whose abstraction is the concept *ACTION). An input string such as He can trigger the propagation of information such as NP[NOM] as well as the semantic propagation through the abstraction and other links. The HMCP models demonstrate that constraint prop­ agation schemes can handle sentential constructions such as control (as in He tried to give Mary the book) assigning the correct interpretations and attaining the correct grammatical­ ity judgement.

4 The lexical entries are originally taken from the representation under HPSG (Head-driven Phrase Structure Grammar, [Pollard and Sag, 1987]). HMCP attains the HPSG unification based analysis under the DMA framework.

3 MSP vs Conventional Architecture

Although the DMA models and the MCP (HMCP) mod­ els are appealing schemes for natural language processing and inference, because these models require propagation of information (i.e., sources of activation and constraint infor­ mation), it has been difficult to support them in the con­ ventional neural-net architecture. In this paper, we refer to the structured marker-passing algorithms that the DMA and MCP models assume in their implementation as 'Massively­ parallel Structured-marker Passing' (MSP) algorithms. Be­ cause the MSP algorithms assume a massively-parallel ma­ chine architecture, the implementations (such as on MULTI­ USP, [Halstead, 1985]) have been generally slow. The con­ ventional massively-parallel machines (such as Connection Machine, [Hillis, 1985]) are not desirable for MSP algo­ rithms because they are incapable of propagating informa­ tion without large overhead. In other words, the massively­ parallel machine architecture, which assumes the propaga­ tion of small amounts of scalar values is not suitable for MSP algorithms.

4 Frequency Modulation Neural Network

We would like to introduce a model of neural network and a possible hardware architecture which directly support the MSP paradigm. We call this model the Frequency Modula­ tion Neural Network (FMNN) and the major features which distinguish FMNN from other neural network are as follows:

• Simulates spikes instead of activation levels.
• Assumes neural clusters with certain circuit topologies: Heterogeneous Neural Network.
• Patterns of spikes, modulated as FM signals, have sig­ nificance in propagating information.

4.1 Simulating Spikes

Unlike most neural networks which simulate the activation level of each node, the FMNN simulates the spikes which each neuron emits. Spikes are electric pulses emitted from neurons to propagate through axons. We are proposing to simulate the neural spikes because (1) we are interested in simulating a detailed biological process, and (2) patterns of pulses can carry information impossible to transmit in past neural network architecture. Biologically, we are not able to define the precise roles of spikes in the human brain with regard to cognitive activities. It is also probably true that mere simulation of spikes will not contribute much to our understanding of the physiological nature of the brain, be­ cause other factors, such as chemical substances, neurogen­ esis, and plasticity of the brain, are also involved. However, we expect that spike simulation may lead to the discovery of new computational features of the brain 5 · Apart from bio­ logical observations, the use of spikes in the neural network makes the flow of structured information attainable.

5 In fact, [Loeb, 1985] has shown that differences between ar­ rival times down to 10 microseconds can be detected by neurons leading to [Sejonwski, 1986]'s analysis that submillisecond timing information could be important in the cerebral cortex as well.

4.2 Heterogeneous Neural Network

The second assumption of the FMNN is the heterogeneous structure. We assume the existence of several types of groups of neurons and certain local and global structures in the network. Most neural networks assume homogene­ ity. Although these models employ certain global structures, such as layers composed of input, hidden, and output units [Rumelhart et. al., 1986] or a recurrent structure [Elman, 1988], each node is assumed to have homogeneous charac­ teristics. In addition, the existence of a local structure is not assumed. Motivations for assuming heterogeneous structure are based on both biological and technical considerations. Biological observations on neurogenesis [Murphy et. al., 1983] [Nottebohm, 1989], the existence of local structure of neurons [Rosen et. al., 1989], and the existence of the specific global innate configurations for the specific tasks such as the Papez circuit involving the hippocampus [Pa­ pez, 1937] indicate that neural circuits in our brain have certain innate structures already encoded in the DNA. In addition, strong evidence of hypercolumn in the A17 field of visual cortex, models of the hippocampus [Zipser, 1985] and studies on the topological circuits of neurons [von der Malsburg, 1985] lead us to assume a structured functional module which we call hypermodu/e. We claim such local circuits provide technical benefits because they will allow us to predefine highly functional neural circuits without aban­ doning the basic features of the neural network.

4.3 Frequency Modulation

We use Frequency Modulation (FM) signals as one of the basic communication mechanisms in our model. When we look into the physiological data of signal transmission be­ tween neurons, frequency of pulses varies depending on the strength of the input given to the neuron. [Treutlein and Schulten, 1985] analyzed the Fokker-Plank equation corre­ sponding to the stochastic Bonhoeffer-Van der Pol (BvP) model, and concluded that the noise level can be employed to tune firing frequency of Hodgkin-Huxley type neurons. In their analysis, the noisy BvP model tends to limit cy­ cle in which the mean period of pulses is depending upon the noise level. This means that the more energy the cell gets, the higher the frequency of impulses it emits 6 · This property has been known since [Adrian, 1946]; however, it has been neglected by the past neural network models. Moreover, there exist neurons whose activation levels vary dependent on the regular brain wave (B-Cells associated with the hippocampal B-wave.). Also, the importance of arrival time of impulses and relative timing of impulses for pro­ cessing in the cortex has been noted (such as [Sejonwski, 1986] and [Sachs et. al., 1983]); however, no neural net­ work model so far has captured this phenomenon 7 · This characteristic is well simulated when we consider FM as its approximation. FM is a kind of an angle modulation technique originally developed for telecommunication of sig­ nals [Marubayashi, 19811. A signal is encoded into a car­ rier wave (Acos(wct +Be)) by modulating its phase angle

6 [Sejonwski, 1981] introduces an idea of skeleton filter - the temporary network of neurons near threshold that can linearly trans­ form correlations between spikes on input lines.

7 0ne of the few exceptions may be [von der Malsburg and Schneider, 1986].

(B = wet+ B c). In the FM, the phase is modulated according to:

~ is called as 'instantaneous frequency' in which it follows:

Since we are going to deal with impulses instead of ana­ log waves, instantaneous frequency is a probability density of the impulse. Apart from physiological aspects, the use of FM signals as representation and communication scheme of a massively parallel computer would provide significant advantages over traditional computer architecture.

5 Modular Neural Circuits in the FMNN

In this section, we describe some of the circuit topologies of modular neural circuits and their behavior. We will be dis­ cussing the circuits which are of particular relevance from the viewpoint of inferencing and natural language process­ ing. Relevant work has been done extensively by [Amari, 19781. A basis of our analysis also assumes [Amari, 1978], althoufh we made some reformulation in order to simulate spikes .

5.1 Simple Random Circuit

A simple random circuit is a neural circuit in which each neuron is connected to the others at random [Amari, 1978]. The external behavior of this circuit is almost equivalent to the McCulloch-Pitt model assumed in most neural network theories [McCulloch and Pitts, 19431. This circuit is also the simplest form of a bi-stable circuit. A simple random circuit stays at the low level equilibrium until input exceeding the threshold comes in. In this case, the circuit will transit to the high level equilibrium. As the signal goes off, the system will go back to the low level equilibrium. The other way to attain a simple bistable circuit is by connecting two excitory neural groups and one inhibitory group. An implication of such a circuit is that this neural complex can act as a node which is commonly assumed in connectionist (such as [Waltz and Pollack, 1984] and [Bookman, 1987]) literature. The circuit is a threshold device and it holds its activation for a certain period of time. A sequence of pulses emitted from such a circuit is random and has no significant meaning as a carrier of information.

5.2 Induced Excitory Oscillator

Induced Excitory Oscillators (lEOs) are types of circuits that generate pulses of a certain frequency whenever a certain ex­ ternal stimulus is given. This is a kind of bi-stable circuit in which one of the attractors is a periodic attractor 9 · An oscillator unit of neurons generates a group of pulses of a certain frequency. Each unit has its own frequency called its 'Characteristic Frequency' or 'Eigen Frequency'. Small circuits consist of a few neurons and with a large number of such circuits with a phased array configuration, output spikes form a simple sequence of pulses with a specific in­ terval. Moreover, a circuit with a group of neurons that outputs a sequence of pulses with a specific modulation is also possible. Such circuits tend to limit a cycle. A spon­ taneous frequency or a probability density of the pulses(Pi) follows:

8 The behavior of the neural circuits in this paper has been an­ alyzed in part by using a neural spike simulator.

9 Biologically, circuits with coherent oscillatory capability are interesting with respect to their relation to Stimulus-Evoked (SE) -resonance of 35-85Hz found in the primary visual cortex called Coherent SE-resonance [Eckhorn et. al., 1988].

In the modular neural circuits, we only require IEOs to sat­ isfy a weak form of the Poincare-Bendixson-Theorem [Bel­ trami, 1987]. The theorem indicates that for the system .X = f(x), x in R 2 , an orbit -y tends to a limit cycle r if -y+ = r and r is a non-constant closed orbit. -y+ is the positive limiting set which is a set of points p in R 2 for which

Our neural circuit is stable if there is a cycle -y which is asymptotically stable. This would require the existence of an open neighborhood .flo such that every orbit starting from .flo has -y as its positive limiting set. This is known as a weak form of the Poincare-Bendixson-Theorem. The circuit that entails limit cycles that satisfy this theorem is useful since it continuously emits characteristic frequency, until the energy is dispersed or inhibition is imposed, which can identify the source of activation.

5.3 Resonance Circuit

Resonance circuits react only to a sequence of pulses whose frequency matches the characteristic frequency of the circuit. Such circuits can be built by combining a series of nodes forming a loop and some inhibitory connections to the input node. When a pulse comes in, the input node gets activated and emits an impulse to the next node in the loop. That node emits an impulse to the next node in turn. At the same time, the node emits an impulse to the inhibitory node in which the inhibitory link is connected to the input node. Thus, the input node can be activated until the inhibition is removed. By adjusting the size of loops and inhibitions, the characteristic frequency of the circuit can be adjusted.

5.4 Sequential Prediction and Activation Circuit

The capability to handle natural language requires the sys­ tem to recognize sequential patterns. This type of nodes is assumed in the DMA type connectionist (or pseudo­ connectionist) architecture (such as [Riesbeck and Martin, 1985] and [Tomabechi, 1987]), in which time-sequenced ac­ tivation of nodes is assumed to model subcategorization of syntactic units, episodic knowledge invocations, etc. In our model, sequential pattern recognition and predictions are at­ tained by assuming a series of interconnected multi-stable modular circuits. As one example of such circuits, we can assume a complex of modular nodes each having inactive, prediction and recognition states. When input signals are less than a certain level (L2), the circuit stays inactive. Once the input strength exceeds that threshold, the excitation level goes up to the pre-excitory level. However, if the input strength is below the excitation threshold (L3), the excitation level stays at prediction level for a while and goes back to an inactive state quickly. The circuit jumps up to a recognition state when the input signals are more than L3. A recogni­ tion state is a stable state, but returns to an inactive state as a result of dispersion of the energy. Inter-module con­ nections are created so as to transfer sufficient impulses to the next module so that the next module can be activated to the prediction state whenever the current module gets to the recognition state. Such network configuration has not been assumed in the traditional neural network. However, we as­ sume the existence of such local circuit topologies in the neural network. The excitation level of the 1-th multi-stable node can be described by:

where £1, S1, uj- 1 denotes the excitation level of the 1-th node, external stimulus to the 1-th node, and excitation level of the i-th element of the 1-1-th node. a1, w:~ 1 i· h denotes a coefficient, weights, and threshold, respectively. The activa­ tion level of the node that packages the series of sequential nodes is:

One interesting feature of this circuit is that it can handle the transposition of activation sequences by utilizing the cost of excitation propagation (instead of not recognizing the in­ correctly ordered sequence at all). To be more specific, a correct input activation sequence will excite Ecsc node at a normal level, say 100. However, an incorrect ordering can exist at a lower level, say 60. This is because an in­ correctly ordered sequence will consume more energy than the correctly ordered one due to the pre-excitation mecha­ nism which carries the next module to the prediction state. One of the reasons that we use groups of neurons instead of asymmetrically connected single neurons is thay they can be combined with resonance circuits to handle context-free rules with some syntactic constraints which are implicitly encoded in form of the modulation frequency. For example, we can implicitly encode the obliqueness order of HPSG [Pollard and Sag, 1987].

6 Knowledge Representation in FMNN

A complex knowledge structure can be built by modulating carrier waves by a modulation signal. Suppose that a cer­ tain group of neurons is representing the cluster for some instance of an already known person such as JoHN, and this cluster of neurons has the eigen-frequency of 200 Hz. Now suppose some activations have triggered the syntactic real­ ization that an input noun phrase that activated the clusters for JOHN concurrently activated another group of neurons that is representing the feature case nominative whose char­ acteristic frequency was 5000Hz. While propagation for the activations of these clusters are performed, the carrier fre­ quency representing case nominative can be modulated by a signal representing JOHN. Such a scheme for modulation is nothing new to the current communication technology as has already been well accepted and performed by FM radios, TVs and other frequency modulated methods of communica­ tion. Intuitively, our concept of variable binding is similar to FM broadcasting in which carrier frequency of 80MHz

(FM-Tokyo) carries the music of the Boston Symphony Or­ chestra (BSO) as a modulation signal (value to be bound). In this case, the music of BSO is bound to FM-Tokyo. A complex structure as seen in Figure-1 (which partially cap­ tures the subcat feature for give as seen previously) can be built by modulating signals.

We assume that one group of neurons is oscillating at a certain frequency. A series of pulses are emitted toward a group of neurons which eventually emit frequency modu­ lated pulses. A modulation signal is given from the group of neurons which oscillate in the frequency representing the value to be bound to the variable. The resulting pulse is modulated based on the frequency of the modulation signal. In case of figure-1, the data structure would be represented in the following equations:

7 FMNN Machine

Building an FMNN Machine should be the best way to take advantage of this architecture. It is analogous to the moti­ vation for building the Connection Machine ([Hillis, 1985]) for implementing the connectionist network. In this section, we discuss some of the possible implementation strategies and outline relevant technologies.

7.1 Processors

A straightforward way of implementing the FMNN model is to build a VLSI neural network chip with heterogeneous connections and spike generation capabilities as described in this paper. One other approach is to build components that are equivalent to each local neural circuit in their functions. Such components would include energy-spike converters, oscillators, resonators, etc .. Several technologies are avail­ able to actually implement such functional modules. Analog circuit technology is a well-established technology that can attain functionalities of intended modules. Especially, recent studies on analog VLSI [Ryckebusch et. al., 1989] may pro­ vide a hardware basis from the VLSI level. Digital Signal Processing (DSP) technology is one other possibility which can be more compatible with current computer architecture. One advantage of using DSP technology is that the filtering and control of signals can be more precise than with ana­ log technology. Also, it is be possible to implement FMNN using current computer technology. However, it would be more likely to be a simulation of FMNN rather than a direct hardware implementation. Figure-2 is a summary of differ­ ences of the FMNN model and other models of computation.

7.2 Communication

In the FMNN Machine, communication between processor units is performed by sending sequences of pulses. The con­ ventional approach of communicating pulses between pro­ cessors is using bus, N-cube, and a network which has prob­ lems such as load bottlenecks, routing, packet collision, etc .. A new approach which we are proposing is to send them us­ ing carrier waves. The pulses are carried on a specific carrier frequency possessed by each generator. Resonators in each node can tune to the desired carrier wave in order to estab­ lish appropriate connections. This communication method enables us to send many independent pulse sequences at a time; thus enabling us to simulate a very large-scale neural­ network (VLNN) in which a massive amount of informa­ tion is transmitted throughout the network. As briefly de­ scribed earler, an FM radio is a good analogy to our ap­ proach, where the signal is sent by a certain carrier wave and an audio signal is modulated by frequency. To send a huge amount of impulses, communication media need to have a large capacity to communicate on many independent channels. Recent advancements in communication technolo­ gies including Optical Wavelength Division Multiplexing or Optical Frequency Division Multiplexing may provide such capabilities.

8 Discussion: Beyond PDP

8.1 Bridging the Gap between Neural Network and Cognitive Processing

The fundamental premise of connectionism has been that in­ dividual neurons do not transmit large amounts of symbolic iriformation ([Feldman and Ballard, 1982]). We are extend­ ing this claim to add a phrase that, however, a group of neu­ rons is capable of oscillating pulses to carry information. As a basis of this extension, we are assuming the existence of local functional circuits which we refer to as hypermodules. The impact of this addition leads to the significant enhance­ ment of the neural network because; (1) we now have a heterogeneous network with highly functional local circuits of neurons that are oscillating, receiving and modulating sig­ nals, (2) the FM signals propagated in the network can con­ tain information such as which oscillating node initiated the propagation and what constraints to inhibit/delay/impose and slow/accelerate certain propagation. Especially the capacity to propagate information results in the capacity to perform operations such as variable-binding (by specifying where the oscillation was initiated) and structure-building (modulating the oscillations on top of receiving frequencies and utilizing the variable-binding capacity). This is significant for cogni­ tive research which requires propagation of information and constraints. Indeed the network is capable of supporting the massively-parallel structured-marker passing (MSP) from the hardware level. As discussed in detail in [Tomabechi and Tomita, ms.J, the MSP network is a model that is capable of performing inferencing and natural language processing that is not currently possible with the conventional connectionist model due to its lack of capacity to propagate information (which is done in MSP as a passing of structured-markers). Also as reported in [Kitano and Tomabechi, msJ, we can build circuits of neurons within the FMNN architecture to capture nodes that respond to certain features of sensorcr input such as sound-wave length and light-wave length 1 · Representation of knowledge in our model is very different from other existing models. Especially, our model does not assume a static representation of knowledge as it has been the case in other models. We are assuming constant informa­ tion flow in the brain (as opposed to the specific portion of the network statically representing some knowledge) and the knowledge is implicitly represented as time-varying signals.

8.2 Physiological Relevance

From the physiological point of view, recent studies of neu­ rophysiology including the one by Eckhom's group [Eck­ hom et. al., 1988] provide some biological relevance to our model. They report stimulus dependencies of oscilla­ tory responses 11 · Moreover, coherence of SE-resonance was found within a vertical cortex column, between neighbour­ ing hypercolumns, and between two different cortical areas. They claim that "SE-resonance can be phase-locked within half an oscillation cycle up to a distance of lOmm at 50Hz." and conclude that coherent SE-resonances are likely to constitute a second higher stage of sensory coding. They further claim that "We are convinced that SE-resonances are a gen­ eral phenomenon, forming the basis of a correlation code which is used within and between different sensory systems and perhaps even throughout the entire brain." These Eck­ hom's findings are consistent with our speculations on the basis of the FMNN model which assumes the significance of resonance frequency created by a group of neurons.

10 Such a property of neurons are known to exist ([Crick and Asanuma, 1986], [Zeki, 1978], [Michael, 1978]), but has not been captured in the past neural network models.

n"The type of visual stimulation was found to influence the frequency of the dominant spectral peak of oscillatory responses." and "The mean frequency of the spectral peak increased by 5Hz if the stimulus drift velocity was doubled."

9 Conclusion

We have proposed an FMNN architecture which is a neural network with a heterogeneous composition and with a ca­ pacity to attain variable binding and structure building using the frequency modulation of pulses by groups of neurons. Assumption of hypermodule allows us to assume modular circuits with sequence prediction and recognition capability. Cognitive processing using the MSP models can be attained by combining hypermodules. In our model, cognitive activ­ ities are performed through the modulation and propagation of pulses in a neural network, whereas the conventional neu­ ral network models have been based on the propagation of scalar values. Although the formulation of the details of the formal character of the FMNN architecture is yet to be completed, the introduction of the hypothesis for frequency modulation of neural activation pulses is a significant en­ hancement to the conventional neural network architecture. Three assumptions we introduced in our model (i.e., spike simulation, a heterogeneous neural network, and frequency modulation) provide our model with an information flow and local functional capability which conventional neural net­ works have not attained. Linkage of distant neural circuit through oscillatory signals is one of the interesting features postulated in our model. This is also interesting from the viewpoint of neuro-physiology supported by the discovery of coherent oscillations [Eckhom et. al., 1988]. We have also proposed that it is possible to build an FMNN ma­ chine and that an FMNN machine would be able to perform tasks such as inferencing and natural language processing that are run under a simulated massive-parallelism in the massively-parallel structured-marker passing (MSP) models at the hardware level.

Acknowledgements

The authors wish to thank Jaime Carbonell, Masaru Tomita, Hitoshi Iida, James McClelland and David Touretzky. We would also like to thank other members of the Carnegie Mellon community for fruitful discussions. Margalit Zablu­ dowski was especially helpful in preparing the final version of this paper.