An Attack on the Problems of Speech Analysis and Synthesis with the Power of an On-Line System

Introduction

The development of interactive software and hardware to permit convenient man-machine commun­ ication has been underway for approximately ten years. Hardware which provides the channels for this communication necessarily relates to man's I/O, his senses. The software supporting this communication must consist of two parts: one which extracts the programmatic implications from the man 's inputs to the computer system and the other which generates outputs perceivable by man's senses. The nature of these outputs depends upon the computer's programmatic perception and re­ sponse to its own internal data, program, and sta­ tus configurations. The use of speech as the mode of communication has become urgently attractive to many of us engaged in building interactive sys­ tems. There are many reasons for this concern: man 's ease in chattering, the freedom to do manual things in parallel with speaking or listening, and the universal character of our existing communica­ tion systems. The required hardware is trivial; it need only consist of a modest microphone and speaker system with an ordinary audio-amplifier which is attached to the A/D and D/A input-output of the computer to be used . The problem lies in the two part software. Here is an outline of our approach to this problem:

WITH THE PCMER OF AN ON-LINE SYSTEM

Glen J. Culler University of California, Santa Barbara Santa Barbara, California

1. ASC¢N WAVE FUNCTIONS

Determine a mathematical formulation natural for the representation of wave forms observed as microphone outputs during speech.

2. ASC¢N TRANSFORM

Develop algorithms requir~d to transform the digital representatio: of the micro­ phone signal into the parameters of the mathematical formulation.

1. ASC¢N INVERTERS

Develop algorithms required to transform lists of mathematical parameters into the digital form required by the D/A which drives the speaker system.

4. PHONE RECOGNIZER

Classify the parameters actually occur­ ing in speech output as functions of the elements of speech.

1. PHONE SYNTHESIZER

Generate parameter strings to drive the output system to produce high quality program controlled speech.

1. LINGUISTIC FORMATTER

Classify phone sequences to determine phonetic, phonemic, and prosodic content and list an encoded representation as a linguistic string.

1. VOICE SYNTHESIZER

From linguistic strings, construct phone sequences to drive the phone synthesizer.

1. LINGUISTIC ANALYZER Extract meaning from linguistic strings.

9· LINGUISTIC SYNTHESIZER

Construct linguistic strings from pro­ grammatic meaning.

At the present time we can announce excellent success in (1.), because we already have suffi­ ciently good success in (2.) and (3.) which is, of course, the first criterion for the adequacy of (1.). We have completed successful experiments in the pair (4.) and (5.) for a restricted set of speech el ements and are currently engaged in com­ pleting this part of our effort. We can give a very tentative and premature definition of "voice" and "linguistic strings" in relation to (6.) and (7.). Only after (4.) and (5.) are complete can we make firm specifications for these definitions. At present, consideration of (8.) and (9.) are be­ yond the scope of our development.

The Speech On-Line §Ystem

The initial hardware complex started with an inter-connection of an early form of the U.C.S.B. on-line system with a process control computer. The configuration consisted of:

1. RW-4oo 1 polymorphic system
2. IB~l8oo 2 mod II process control computer

1 The RW-4oO system was contributed to U.C. S.B. by the Bunker-Ramo Corporation for further investigation of on-line systems.

2 The IBM-1800 system was made available on our ARPA project, Contract # AF 19(628)-6004 by the Department of Electrical Engineering at U.C. S.B.

1. Tektronix 611 3 , 11" storage tube
2. Microswitch 4 operator-operand keyboard

which were combined as shown in Figure 1.

The first form of our speech station was re­ ported in [1. J at "The Conference for Interactive Systems", August 1967. The principal hardware change since that time has been the addition of Bill Proctor's "Nanohumper"*, which is the right way to tackle the problem of broad band A/D con­ version. Our major task has been the experimental construction of software required for the investi­ gation of (1.) and its validification by (2.) and (3.). Roughly speaking, it consists of a set of operations which process a long data list and a set of others which process a long parameter list in the process control computer which is super­ vised by the early U.C.S.B. on-line system run­ ning in the RW-4oo system. For a detail ed de­ scription of this system and its operations, the reader i s r ef erred to the Final Report ARPA Con­ t ract # AF 19(628)-6004.

3The Tektronix 611 was donated to U.C.S.B. by the Tektronix Corporation to provide early experi­ enr.e in its use.

4 The operator-operand keyboard was developed at u.c.s.B. under ARPA contract # AF 19(628)-6004 as the input portion of a classroom system.

* Records time and voltage of fixed voltage level transitions. Thus the volume of data output is proportional to frequency rather than time.

Elementary Sounds

After some initial investigat ion of the in­ adequacies of both Fourier series and transforms as mathematical frames for representing the vol­ tage signal output by a microphone during the speaking process, we turned to wave function re­ pr esentations of limited time duration . When one of these wave functions is given as the specified voltage used to drive a voice coil on an audio speaker, the result is an "elementary sound". Sounds of arbitrary complexity can be constructed by superposition of t hese elementary sounds. The extent to which all speaking voices can be dupli­ cated in this manner is a measure of the compl ete­ ness of these sound elements as a basis for speech. The simplicity of the resulting represen­ tations of live speaking voices in terms of ele­ mentary sounds is a measure of the wisdom of se­ lection of that particular basis, for simplicity of representation carries continuing value as we move up the ladder of efforts outlined above . The parameters used to characterize the el ementary sounds can be motivated by considering the pres­ sure wave function. This pressure wave is a dis­ turbance propagating through the medium surround­ ing the speaker and can be recorded by a standing observer with a microphone. Assuming the speaker and microphone are of excellect fidelity, we can thus obtain an experimental r ecording of an ele­ mentary sound. In the coordinate system of the observer, the disturbance will have an amplitude A, will last for a certain time span S, and will have associated with its occurrence a center time C half way through the disturbance. These are called the global parameters of the elementary sound, since they depend only upon its traveling envelope. Now, within this traveling envelope, we characterize the pressure oscillations by N and ¢, where N i s the number of oscillations recorded during the time interval S and ¢ is the phase of the internal oscillation at t = C. As a means to

A amplitude

s

c

¢

time span

traveling center time

oscillation phase

N oscillation order

Figure 3. Graph of pressure wave of an elementary sound with its ASC¢N parameters.

familiarize the reader with the visual aspects of our wave functions and their ASC¢N parameters, we invite you to try estimating all par~~eters for those wave functions shown below. Correct answers are given at the end of the paper, A in volts, S in milliseconds and¢ in radians.

Elementary strinf~: treble, mid, and bass

The speaking vr~ ces of man are so p~rsonal­ ized and rich in frpm ren ~y rv.nge that one despairs early of hoping to c ~r<J.cterize speech in terms of statical averages o_ frequences, extr emes , zero crossings, etc . Instead, we elect to carry out a detailed reduction of each speech expression to the elementary sounds of which i t i s composed, thus deferring sound recognition and interpreta­ tion processes as shown in the outline above. This is the purpose of the ASC¢N-transform of (2 .). With classical transforms that are defined as mathematical processes, one can always state a transformation formula in explicit terms . Doing it is often the most distinctive of the three. The mid string i s the most familiar of the three in terms of ·voice; when combined with the trebl e, it makes a voice much like that r ecorded in an anechoic chamber. The bass is not often of dis­ tinctive importance; it represents much of the acoustic interaction of the room, the voice, and 1 the recording system. In our on-line speech sysj 1tem we have included the facility for programma­ tically constructing digital filters. Since we are still in the study phase, we have the freedom I to carefully observe a sampl e word and experimen1tally select a means of separating out the strings. In crude terms, the treble is everything above 1800 Hertz, the bass is everything below 450 Hertz, and the mid is everything between. A good 'test for completeness of separation into strings is that the second derivative of each of the three must remain in the same category as the original string, or the separation is incomplete.

Each string is then decomposed into its con­ stituent elementary sounds by the method of evacu­ ation, i.e., first remove those of primary influ­ ence from the string by insisting that after re­ :1 moval of a wave function, the residue string is of minimum amplitude in the neighborhood of the one removed. After each evacuation the residue is a similar string of smaller amplitude, and the pro­ cess can be repeated. In Figure 5. we show the three strings in the decomposition of the signal produced by a male voice saying "Corby". As is apparent from this figure, the fre~uency is not constant across a string, but when properly mea­ sured for individual elementary sounds will pro­ vide an important parameter for sorting these sounds. The evacuation method is shown in Fig1 ure 6. for the mid string of "Corby" during the ~ first 210 ms . Figure 7. shows the sum of the first two evacuations in comparison with the original mid string.

Figure 4. Wave functions of some typical elementary sounds in their envelopes.

so, however, does not remove the need for an al­ gorithmic computational method, since these for­ mulae fre~uently are not formulated for computa­ tional re~uirements but rather for descriptive mathematical expression. In the case of our ASC¢N transform, we can state an algorithmic method for computing the transform, although no mathematical formula is known. The computer effort re~uired to carry out this algorithm is greatly benefited by some initial reduction of complexity in speech da­ ta. For us, this reduction consists of decompo­ sing the speech data into strings of elementary sounds of separated fre~uencies. These strings are in rough agreement with the classical selec­ tion of formants, although they occasionally dif­ fer. We distinguish three strings: treble, mid, and bass. In some sounds one or another of these strings may have very low amplitude, and with dif­ ferent sounds one or another may assume major im­ portance. In general , the treble string variation indicates most sharply transitions in sound; thus Each elementary string f(~) can be expressed as a sum of el ementary sounds through the evacua­ tion process. This sum takes the form where E is computed from the wave e~uation:

Part (3.) of the outline is thus a computational analysis of how to solve 4.2) in real time without introducing undesirable computational anomalies. One can experiment with many methods and observe their shortcomings by listening to 4.1). This way we can frequently hear troubles which are diffi­ cult to detect by ordinary means. We are still working to improve our methods here and produce a hardware specification for a "string" instrument, i.e., a special-purpose module which can read a string of ASC¢N parameters and produce the real time sum 4.1). Three speech strings are capable of producing arbitrarily good speech.

Phones, The Simplest Structural Elements in $peech

The transformation of digital sample data to elementary strings and finally to lists of elemen­ tary sounds represented by lists of ASC¢N parame­ ters puts us in a position to study these lists and determine which subsets have sufficient inner cohesion to be distinguished as speaking sounds. In Figure 8. we show A, s, ~C, ¢, and w plotted against the index j for a single mid string recor­ ding of the word "odd';. · Clearly, what catches our eye (and our ear as well) is the modularity of the above data and an apparent regularity of the evolution of the parameters up to ragged changes, at which time other trends set in. Thus, to isolate the simplest substructures of these parameters, we base the selection on the smoothness of variation of the parameters. We then define:

• A phone is a set of non-overlapping elemen­ tary sounds with smoothly varying ASC¢N parame­ ters.
• The pitch function for a phone is the or­ dered set of time displ acement inverses, {(~cj)- 1 , j = 1, ... , t}; the pitch is the mean value of the pitch function.
• The frequency fUnction for a phone is the ordered set {(Nj/Sj) · 1000} (in Hertz); the frequency is the mean value of the frequency fUnction.
• The span function for a phone is the or­ dered set {sj}; . the span is the mean val­ ue.
• The phase function for a phone is the or­ dered set {¢j}; the phase is the mean val­ ue of the phase function.
• e . The amplitude function for a phone is the ordered set {Aj}; the amplitude is maximum value of the amplitude function.

The condition of non-overlapping elementary sounds in phones is too stringent for some of the consonants whose elementary sounds have broad spans in the bass string and is too l enient for ·vowel s, as we do not want to allow more than one elementary sound in the same phone per pitch period. The pitch period of an elementary string is determined by the pitch fUnctions of the principal phones, those of largest amplitude in the string. The principal phones of the first 210 ms ·. of the mid string of "Corby" are shown in Figure 9.

Voice; The Comprehensive Structure of Sound in Speech

The concept of a "phoneme", that which mini­ mally distinguishes between two words, leads us to a study of complexes of phones. strictly speaking, a complex of phones that in a given utterance operates according to the concept of a phoneme comprises a phonetic realization of the phoneme in question.

Our tentative definition of voice is a set of prescriptions of phone complexes that give rise to phonetic realizations of the speaker's phonemes. The expression of voice, sometimes called prosodic control , is a transformation of the amplitude and pitch of these phone complexes just prior to final construction of the phonetic realizations. That is, the voice consists of prescriptions for stan­ dard phones (unit amplitude and unit pitch) which are modified by the prosodic controls. To extract the voice from input data, the speaker may recite some well-chosen words containing the presumed phonetic realizations of different phonemes (cer­ tainly with cross-checking to diminish confusion). The resulting phone complexes must then be sim­ plified by smoothing and stored in the voice table in a way that is advantageous for comparisons . This is a two->vay table associating phonetic sym­ bols with phone complexes . One way provides a ba­ sis for analysis, the other way for synthesis .

In summary, whi l e there still is much diffi­ cult and challenging work to be done, we have sue­ ceded in constructing thoroughly tested foundation elements which permit us to record accurate and simple mathematical parameters of speech sounds which provide high-fidelity reproduction and which open the way to future, perhaps general , computa­ tional analysis and synthesis of speech.

Acknowledgements

The work reported here was primarily support­ ed by the .Advanced Research Projects Agency, Con­ tract # AF 19(628)-6004. A number of dedicated co-workers have applied themselves in an extraor­ dinary way and have contributed broadly to our present success. We are especially indebted to Gordon Buck and Helen Smith for their programming success; John Greaves, · Gary Nelson, and Bill Proc­ tor as graduate research assistants; Ray Bjorkman, Gordon Buck and Dennis Grubbs for their hardware design and construction; Professors Jim Howard, Roger Wood and their students for their continuing experimentation with our underlying engineering processes. Finally, I would like to thank the ARPA personnel, formerly Ivan Sutherland and pre­ sently Larry Roberts and Robert Taylor for their encouragement and stimulations to undertake this research.