An Associative Memory for Auditory Recall

AN ASSOCIATIVE MEI'KJRY FOR AUDITORY RECALL

Richard J. Reid

Computer Science Department Michigan State University East Lansing, Michigan U.S.A.

Abstract. A model of auditory temporal recall based on correlographic recording and inter­ rogation by convolution is presented. Complex tree and loop structured memory sequences can be recorded, with selective recall based on fragmentary cues. The sequences are "re­ membered" by proceeding through a recorded net of associations until the coherence of the retrieved signals falls to a sufficiently 19W level. This indicates no further significant recall is possible without an additional exter­ nal cue.

A computer simulation of this model using digitized speech has produced variable-length and cyclic sound-segment recall from fragmen­ tary cues.

Introduction. The problem of recording infor­ mation in a manner so as to associate it with other recently received information and/or old information recalled by the present environment, appears to be at least a prerequisite to intel­ ligent behavior.

We are concerned here with presenting a model of paired-associate recording and recall that may have the necessary potential.

The possibility of recall based upon holo­ graphic techniques was presented by Stroke (1) as a compensation or filter effect, and later by Collier and Pennington (2) as recall based upon fragments (possibly displaced) of the orig­ inal scene. This verified Van Heerden's (3) earlier prediction made on a theoretical basis.

The experimental setup for displaying this phenomena is shown in Figures 1 and 2.

An interesting possibility of using this as a model of an associative memory comes from the fact that multiple exposures can be made on the same hologram and, after development, illum­ ination by a fragment from one of the scenes will recall that original scene provided suit­ able circumstances are insured.

In the following dis cuss ion we shall be primarily concerned with auditory signals rather

--!--

Figure 1.

Recording the paired association from a scene composed of two fragments.

Reconstruction of original scene by fragmentary cue from the original exposure.

than visual, but thay are generally just a re­ striction to one dimension rather than two.

The possibility of recording the auditory analog to Figures 1 and 2 was suggested by Longuet-Higgins (4) and extended to a neural model by Chopping (5). In the auditory model a given time signal is considered to be made up of two successive fragments, a(t) and b(t). Then the magnitude of the power spectral density is · recorded as

where

* is the Fourier transform of a(t) and ( ) is the complex conjugate.

Then upon recall the illumination by A

would give

If we examine a single one of these four sum * terms in Equation [1] , say ABA, it represents the convolution of t he signal b(t) with the autocorrelation of the signal a(t). Assuming the autocorrelation peak at t = 0 is much great­ er than at any of the other registrations, then the convolution of this with b(t) will reproduce b(t). Further assume that for a term such as * AAB , there are no peaks in the cross correla-* tion represented by AB that are comparable to the autocorrelation peaks, In this case, the convolution with a(t) would act to severely

attenuate a(t).

For the above assumptions then, the four terms in Equation [1] give the approximation

where n represents noise as an attenuated and

multiply-echoed a(t ) .

The reconstruction sequence above does contain the recall of b(t) and by a property of the Fourier transform, it will have the same relative time displacement from a(t) it had in the original exposure.

Thus, by recording the power spectrum of the original sequence and illuminating that recording with the spectrum of a fragment and inverse transforming the transmitted spectrum, and finally, playing the resulting sequence, one hears the recall fragment followed by the remain­ der of the original sequence.

It is similarly possible to use the cueb(t) to recall the entire sequence. If the simula­ tion is performed using the discrete Fourier transform, then the multiplications of the spec­ tra correspond to circular correlations or con­ volutions. Thus, in the replay, the recalled segment a(t) would follow after the cue b(t).

Multiple exposures can be made and each exposure can be composed of more than two frag­ ments. However, there is some difficulty in overcoming the noise that is added from these additional terms. For example, suppose we have made two exposures, the first using the frag­ ments a 1 and a 2 and the second using a 3 and a 4 . Then upon attempted recall using a 4 we obtain where then's represent the noise contributions, and, of course, the a 4 's and a 3 have the imper­ fections caused by their recalling autocorrela­ tion terms. But, in any case, the recalled seg­ ment a 3 is present and ~perimentally is discernable.

A study of the signal-to-noise ratios that can be expected based upon various numbers of exposures and fragment length to total signal length has been reported by Willshaw and Longuet­ Higgins (6).

Proposals have been made by Gabor (7) and Willshaw, Buneman and Longuet-Higgins (8) to eliminate a significant amount of noise by re­ cording only the necessary correlographic terms rather than the power spectral density. This

enhancement is included in the sections below.

The very peaked nature of the autocorrela­ tion assumed above does not occur in practical signals, especially speech signals. And, al­ though some have assumed that visual scenes will have this property if they are composed of very narrow line segments, this will insure that property in only one direction. The correlation can still be quite broad for translations along the direction of the lines and may give severe undulating ghosts in those directions.

The next sections also include a method for sharpening this autocorrelation peak without having to restrict the nature of the primary in­ put signals.

A form of sequential recall was proposed by Longuet-Higgins (9), but it is not possible in holographic recording without recoding the info~ mation between successive cues.

The model to be discussed will allow the sequences to be tied back upon themselves to give continuing circular recall. In the simu­ lation, an estimate is made of the signal-to­ noise ratio, and as this falls to a small value, the recall is terminated pending the next exter­ nal cue.

Sound Segments. Experimental facilities have been established for obtaining digitized speech. This is accomplished by low-pass filtering the speech input to less than half the sampling fre­ quency (to prevent aliasing). This filtered in­ put is then sampled periodically and converted via an analog-to-digital converter and stored in digitized form.

To minimize the amount of digital storage required, various sampling rates and numbers of quantization levels have been considered. After some experimentation, a sampling frequency of 4000Hz. and three bits of quantization were adop~ ed as nearly the minimum quality suitable.

After the speech signals were in digital form they were disected into elements for a cata­ log of sound segments. These segments included s poken digits , phonemes and some complete words. These sound segments can _be referenced by a dic­ t ionary process and concatenated for subsequent pr ocessing or digital-to-analog conversion,

Using this standard set of auditory signals obv iates some of the severe problems dealing with the variability of actual speech such as changes in amplitude , pitch and time registration. Al­ though a neuronal model has been proposed by Roy (9) f or accommodating these variations, it has not been included in the present work.

Correlographic Recording. The paired-associate storage mechanism makes use of the discrete

to transform the digitized speech segments x(k) to the frequency domain X(j). Each segment is padded to be 4096 samples long (one second of speech) and the fast Fourier transform is used,

Associates come in pairs, but they each re­ present a separate one-second sound interval and they are transformed separately. Their associa­ tion is recorded as the product of the conjugate of the first transform and the second transform, and this product is added to any previously ac­ cumulated correlographic sum as

If the above second sound segment were t o be followed by a segment c(t), then c would be sep­ arately transformed, associated with b and accum­ ulated as

This sequence of paired-associations can be continued down a chain of successive speech seg­ ments and/or new paired-associate sequences can be added, For example, the correlogram

records the associations a-b, b-e , and d-e.

The individual paired-associates are equi­ valent to the discrete Fourier transform of the circular cross-correlation between the sound seg­ ment pairs, i.e.

where k is the index of the transform summation, DFT is the discrete Fourier transform, and the index (l+k) is evaluated modulo N.

Recall. Unlike the bidirectional recall proper­ ties of the holographic recording, the cues in correlographic recording will recall only their successors. This directional characteristic has been likened to human recall which can go through the alphabet, weekday names, etc., readily in the forward direction but poorly in reverse.

This recall is accomplished from a cue by taking its transform, passing the transform through the correlogram (multiplication), and inverse transforming the result, For instance, assume the previous recording given in Equation [4], then using b as a cue gives

subject to the previous assumptions about the auto and cross-correlations. Here then, the desired recalled element c may be obtained in a relatively unencumbered manner as opposed to the recall of a 3 given in Equation [2].

The recalled element in [5] was obtained as the convolution of the segment c with the auto­ correlation of b, i.e.

where k is the summation index of the transform, *is the correlation operator, and the indices (k-1) are evaluated module N.

It should be noted that the correlations and convolutions could be carried out in the time domain, but it is about an order of magni­ tude faster to carry it out in the transform

plane in digital simulation when N = 4096.

In dealing with speech signals it is true that the autocorrelation is not a single sharp peak at t·O. Because of the periodic components of speech signals (the periods vary, but rela­ tively slowly), the autocorrelation has a multi­ tude of relatively high and somewhat broad peaks. In the convolution then, each of these peaks con­ tributes a copy of the convolved signal, with each copy delayed by the interval between the autocorrelation peaks. The reconstructed signal is thus highly reverberative and is not easily

identified.

If this phenomena is viewed in the frequency domain, then a sharp single peak in the auto­ correlation produces a relatively uniform spec­ trum which then fully illuminates the recall el~ ment spectrum. However, the multiple-peaked autocorrelation produces more of an irregular "comb-like" spectrum, which only partially (at certain frequencies) illuminates or filters the recall element spectrum. This gives the recalled sound the characteristic that it has been passed through a drain-pipe acoustic filter.

The desired autocorrelation properties can be produced by the following techniques,

In order to avoid the successive reinforcing registrations of the periodic components during autocorrelation, their periodicities must be re­ moved during the recording process. This can be done by permuting the digital samples within a sound segment, and if the permutation gives the signal a sufficiently noise-like character, the autocorrelation will produce the single peak at t=O, and the associated spectrum will be rela­ tively uniform. The reverse of this permutation can be carried out on the recalled element to restore its periodic properties.

This permutation coding seems rather simi­ lar to the first layer transformation in a multi­ layer perceptron as discussed by Minsky and Pappert (10), and the cortical layered calcula­ tions proposed by Kabrisky (12).

The following permutation is easy to visual­ ize and implement and does not require a permu­ tation table. Given an original sample of N = 2m points, and suppose we are going to take points from this sample but that our selector runs at linearly increasing tempo. Then the i'th origi­ nal sample goes into the bj'th cell as

and b 0 = 0. Now if cj equals bj modulo N, then this redistribution of the data is a permutation if

This can be shown to be true.

While the above, or a random permutation, does give the desired autocorrelation result, it is at the expense of a rather desirable fea­ ture of the original recorded transforms. Pre­ viously, if a cue had been displaced from the registration it had during recording, the recall phenomena would still be intact although the re­ called fragment would have been similarly dis­ placed in time. For a cue displacement of 1 sampling intervals

resulting in the autocorrelation spectrum

i.e., the same spectral magnitude for illumina­ ting the fragment to be recalled, with the phase change causing a unt"form time delay in each of the spectral components--thus the delay in the recall,

With an arbitrary permutation encoding, it is not sufficient to be quite close to the regis­ tration, the registration must be exact.

A further desirable property of possible improvements to this first level encoding should be mentioned here. In order for the various noise terms to be minimal, it is necessary that the cross-correlation between recorded signals have quite low peaks, by comparison. In a par­ ticular sample, over the range of spoken digits, phonemes, and several short words, the ratio of autocorrelation peaks to maximum cross-correla­ tion peaks varies from three or four-to-one to 50-to-one, Of course, this depends greatly upon the samples, and the time and amplitude quanti­ zation. The important point to be made here is that the permutation coding discussed above was done solely to enhance the autocorrelation prop­ erties, while the cross-correlation properties are hardly affected. It may be true that both properties cannot be further improved unless the permutation encoding is dependent upon the signal itself.

Seguential Recall. The possibility of recalling a list of items occurs when one considers using a recall production as a cue for a further inter­ rogation. This is a possibility in correlo­ graphic recording whereas it is not with the hologram or holophone because they reproduce the cue as part of the recall which could not then be directly recycled for further recall. Time division selection also would not be satisfactory in a discrete system because of the periodicity requirement.

Now consider the recording

and an interrogation with the sequence a which produces

If this result is used as a cue for an additional recall we obtain

or

There are several important considerations in this second result, First, consider the pro­ duction of the desired sequence c from the * * term X X x_x_ X · We depend upon the cross-cora a-o~o c relation of a*a and b"'b (or the convolution of a)l(a and ~b , but they're the same in this case) to still produce a relatively uniform spec­ trum for illuminating X · That is, holes in c* * either spectra, that of XaXa or ~~ , are missing from the illumination of Xc.

Next, consider the noise term n 2 which is * * produced by X X x_x x_ · Alternatively, if we a a-o a-o introduce the convolution operator,~ , the time domain equivalent is

Unfortunately, the convolution of the two same cross-correlations tends to enhance the peaked nature of their convolution. This adds more co­ herence to their (the peaks') selections of timedisplaced copies of the signal a Fortunately, the multiple peaks are not conglomerated, although frequently one is somewhat larger than the rest at the outset.

It is interesting to note that frequently the cross-correlation between any a and b as above, enhanced by the first convolution, is sufficient to produce an intelligible n2 ~a if no other components are present. This is true of the n 3 term also, and it is only the re~ atively strong recall of c that masks this weaker secondary effect.

If variable length sequences are recorded, then it is important to be able to recognize when the recycling of productions as cues is to be terminated, so one can return for an external cue for the next interrogation. considered below. This will be

Cyclic Recall. If sequences can be recorded and retrieved as consecutive elements, then the se­ quence could also produce an item that was used as a cue somewhere previously in the list, This will then allow a loop through this sub-sequence. As the simplest example consider

which is cyclic if initiated by either the cue a or b · If furnished the cue a, the r'th recall will produce Again, some detection as to when the re­ cycling is to be terminated is implied.

These cyclic sequences can be longer and they may be preceded by a linear (non-cyclic) preamble.

Composite Recording. If the various possibili­ ties of the preceding sections are combined, the composite correlogram corresponding to the sequential patterns of Figure 3 is possible.

Each of these paired associations is recorded individually, and the order is immaterial; al­ though, in the simulation it is convenient to give sequences rather than individual pairs.

An interesting property of how a single new association can modify the overall behavior is illustrated by considering the recording shown in Figure 3 before the association n-k * was recorded (N K added to the correlographic accumulation) versus the behavior after this single addition,

Termination Condition. With the provision for variable-length and cyclic sequences, it becomes necessary to determine an ending point for the recycling of productions as cues. For non-eyelie sequences this should happen after the last element is recalled. Last elements could pos­ sibly be especially marked for recognition as such, but that would not solve the cyclic ter­ mination problem. Instead, we attempt to use every production as a cue and notice that its production may be totally noise-like, This ap­ proach has been somewhat successful in simula­ tion and proceeds as follows.

After a particular production is available, that is, the inverse transform of the convolution has been performed, the recalled sequence must be permuted in an inverse manner to the encoding permutation. Then the spectrum of this correct­ ly sequenced recall segment is examined for con­ centrations of pcwer which would indicate the presence of a coherent signal. If this spectrum is given as

then define

as the average spectral power, and

as the standard deviation.

Then the original signal is judged to be other than noise if at least several of the * X(j)X (j) terms are more than a few standard deviations greater than the mean, In the pre­ sent experiments with 4096 spectral components, six were required to be greater than five stan­ dard deviations from the mean.

AJ; might be expected, this detection is rather imperfect, and is set to accept possibly several recalls of noise rather than interrupt a legitimate sequence, An improvement would un­ doubtedly result if small clusters of exception­ al power were identified, For cyclic sequences the same termination mechanism is employed, This works because there are more and more convolutions of auto­ correlations convolved with the desired sequence in one particular term of the recall sum and the number of noise terms is increasing. So, as contrasted to non-cyclic recall dropping off the end of the initiated sequence, cyclic recall fades into the noise and is ultimately aborted.

Typical Correlogram Composition, The entries in Table 1 indicate the present range of sequences that can be recorded on a single correlogram and give intelligible recall.

Table 1.

Typical lengths and numbers of sequences and cycles that can be recorded on a single correlogram and give intelligible recall.

The sound elements are each of one second duration and correspond to 4096 samples of 3 bits each, The correlogram is recorded as two real values and 2047 complex values where the expo­ nents are 11 bits and the mantissas 37 bits long.

Summary. It has been shown that it is possible to record several sequences of aural information in a single correlogram and retrieve the individual sequences by furnishing the appropriate cues.

The variable-length sequences are "re-mem­ bered" by cycling a production as the next cue until the output contains no further signals.

As can be seen from Table 1, the present aimulation is rather limited in terms of the total amount of information that can be recorded before the noise produced masks the desired re­ call elements. A practical improvement in this aspect can probably be obtained by filtering the output to consist only of the same exceptional spectral components that are used to identify the fact that there is actually a signal present.

Further improvement may be made by making the permutation (or other) encoding dependent upon some well-preserved--preserved through re­ cording and recall--property of each individual signal.

Of great interest is haw the memory mech­ anism can be made more dynamic. That is, succes­ sive recordings should be made as associations with recent external inputs ~ the internal re­ call they may have triggered. Also, recall it­ self should be done so as to modify the infor­ mation retained.