Time-Domain, Digital Segmentation Of Connected Natural Speech

I. Introduction

This paper describes a segmentation algorithm which forms part of a recognition system for natural speech on the basis of phonemes and phoneme-like elements (Fig. 1). The topic of this paper will be confined to the segmentation steps; the rema1n1ng steps of the system have been described elsewhere /Hess 1974.1,1974.2,1972/.

The extreme difference between the information con­ tent of the acoustic speech signal and its written counterpart forms one principal problem of any automatic speech recognition system. The classifier itself which classifies the signal into the desire~ output classes (such as words, phonemes etc.) usu­ ally cannot cope with an information content too large. Therefore,it needs a preprocessor which re­ duces the great redundancy of the signal. For this, one has to extract a series of significant parame­ ters which, whatever they are, maintain most of the (phonetic) information significant for the classi­ fier, but throw off a great part of the signal re­ dundancy. Another problem in recognition of contin­ uous speech is given by the fact that the output of the recognizer has to be discrete in time, and that the classifier can process a limited number of out­ put classes only. Thus, it is obvious that the out­ put of the classifier in a recognition system for continuous speech cannot be words and even not be syllables /Olson, 1967/. Recognition on the basis of phonemes or similar elementary units, however, requires· time localization of these units in the speech signal. That means, it poses the problem of segmentation. It is left to the processing strategy whether segmentation is done together with classi­ fication (recognition) or during the preprocessing step /Paulus 1974/. In the system described here, segmentation is performed as early as possible.

The first approach to segmentation is already found in vocoder speech transmission systems. This step which, . in the following, is labeled "pre-seg­ mentation", determines two binary features of the

TIME-DOMAIN, DIGITAL SEGMENTATION OF CONNECTED NATURAL SPEECH

Wolfgang J. Hess

Institut fur Informationstechnik, Technische Universitat MUnchen Munich, Fed.Rep. of Germany

• X: This part of the system is described in detail in this paper.

speech signal, both of which are concerned with the voice source:

1. Discrimination "Pause/Speech".
2. Discrimination "Voiced/Voiceless".

Regarding the vocal tract instead of the voice source will lead to a different kind of segmenta­ tion. The vocal tract tries to realize the appro­ priate target position for every spoken phoneme /Fant and Lindblom, 1961; Flanagan, 1972/. For the further investigations, one has to issue from the following assumptions /Bhimani, 1963/:

1. When realizing a spoken phoneme, the vocal tract first will adjust itself to a target position, then will remain there for a cer­ tain time, and finally, will move to the target position associated with the next phoneme. For this reason, fast movements of the vocal tract (leading to "dynamic" or "transitional" segments in the signal) and periods of little change in the vocal tract position ("stationary" segments) will alt­ ernate.

1. Every significant articulation change will cause an essential change in shape of the speech signal as well as its main para­ meters and vice versa.

A spoken phoneme thus consists of a dynamic begin­ ning, a stationary middle part and a dynamic end which is usually j oined by the beginning of the next phoneme to f orm one dynamic transition (Fig. 2).

There are exceptions: diphthongs may contain sev­ eral stationary segments, whereas stops may not contain any stationary part at all. Thus, if one succeeds to locate these stationary and dynamic segments, he will be able to locate the spoken phonemes in the speech signal. The third and main step of segmentation therefore will be described as follows:

1. Separate stationary and dynamic segments in the voiced sections of the signal (and, a­ part from that, also in the longer voice­ less parts whi ch may contain more than one phoneme). ·

In the following, these steps, to which a correc­ ,tion step is added, are discussed in some detail.

2. Pre-segmentation

This step deals with properties of the voice source. It subdivides the speech signal into voiced sections, voiceless sections, and pauses. The methods applied for this step depend on vari­ ous environmental conditions, such as the signal­ to-noise ratio or the purpose for which the al­ gorithm is to be used. For the procedure described here, the results are used in a speech recognition system, so that a fairly accurate knowledge about pauses and speech sections is required. Further­ more, noise level can be assumed to vary slowly with time. For this reason, a fixed level thres­ hold, as applied e.g. by REDDY /Reddy, 1966/, did not prove sufficient for an accurate pause-speech discrimination. Instead, the level distribution of the signal is taken in form of a histogram during preprocessing (Fig. 3). At the noise level, this histogram shows a distinct peak. Since the signal level during speech sections is subject to much greater changes than the noise level, a threshold LPS can readily be derived from this peak; thus, pause and speech sections are separated. To pe r ­ form a better discrimination of the weak fri ca­ tives, this procedure is also applied to the level Ln of the digitally differenced speech signal. By t~is, a second threshold L is derived. Hence, a signal section is classifi~R as "pause" when the levels measured are situated below both of these thresholds:

Since the level parameters LS and L as the absolute average of the signaY differenced signal: are computed an resp. t he

N covers a period of 25 msec resp. - after pitch determination- one pitch period.The level d i str i ­ bution is continuously updated; the influence of more recent values is emphasized by multiplying the whole histogram by a constant less than one from time to time.

Separation done with and Ls: of voiced and voiceless sections i s regard to the ratio QS of the lwels LD

This parameter gives a crude estimate of the be­ haviour of the signal in frequency domain; by fixed thresholds the signal is divided into sec­ tions labeled "voiced", "voiceless", and "may-be­ voiceless". The latter sections will be further classified in the subsequent pitch determination step /Hess, 1974.1/.

3. Choice of Parameters. Segment Length Function. Minimal Segments.

Are the stationary segments sufficiently station­ ary resp. the dynamic ones sufficiently dynamic to be determined by an automatic classification algo­ rithm with reasonable reliability? What parameters have to be selected to perform this? In this step, the feature "stationary" or "dynamic" is not a physically determinable characteristic of the speech signal as e.g. the existence of pitch. For this reason, the parameters selected for segmenta­ tion have be treated in a nonlinear way to dis­ criminate the signal into the desired segment clas­ ses; this can be performed by an adaptive or by a rigid algorithm. As former experiments show, espe­ cially by REDDY and VICENS /Reddy and Vicens, 1968/, the segmentation of the signal into sta­ tionary and dynamic segments seems possible using few global speech signal parameters and a rigid, non-adaptive decision-tree algorithm.

In the following, the three parameters defined in eqs. 2 and 3 together with the binary features ex­ tracted in the pre-segmentation step are used. These parameters are measured pitch-synchronously /Hess, 1972/ and then interpolated at constant in­ tervals ("microsegment interval"). If pitch is not available, these parameters are measured for a pe­ riod of 25 ms before interpolation. The microseg­ ment interval has been set to 2.5 to 5 ms for the speech material used in these investigations. This short interval was selected in order to process even fast transitions in a correct way; this seems justified by the accuracy of the pitch-synchronous parameter measurement.

Concerning the strategy of segmentation, it proves advisable to give priority to the determination of the dynamic segments. As experiments e.g. by OHMAN /Ohman, 1962/ show, dynamic segments are more important for speech perception than stationary ones. Secondly, in a recognition system using the results of segmentation, undetected phoneme bound­ aries will cause irreparable recognition errors. For this reason, the following procedures deal with the dynamic segments primarily.

For further processing, assume the signal to be subdivided into microsegments of fixed length. For each parameter,the relative change between two - not necessarily adjacent - microsegments Mi and Mk is determined as follows:

In this equation,P represents the segmentation pa­ rameter regarded. Note that all the parameters Ls, LD, and QS can take positive values only.K is a correction factor whose influence will be discus­ sed later.The relative change r·k is defined to be a "major" change when its absol~te value exceeds a given threshold qp:

n

TI

qp

r .

nJ

Parameters

Initial and final points of minimal segments

Location for which SLF is computed.

Tolerance interval for lr ·l<qp nJ Sensitivity threshold

Parameter change, as defined in eq. 4

In the figure: SLFn = I

This equation defines the "minimal segment" as se­ quence of adjacent microsegments: its length is determined in a way that between its initial and final point the controlling parameter P is subject to one major change (Fig. 4).

Direction of processing plays an important role in the computation of thP minimal segments, when this computation is performed time - sequentially

P Parameter

TI Tolerance Interval

I Forward Processing

2 Backward Processing

Fi15. 6: Primary Segmentation.

STA Stationary

DYN Dynamic p Parameter qp Sensitivity

UND Undefined threshold

sST SLF Threshold "Stationary"

sDY SLF Threshold "Dynamic"

(see example in Fig. 5). The influence of the pro­ cessing direction causes significant errors and, for this reason, has to be eliminated. That means that th~ minimal segments cannot be computed time­ SE'<(Uf'ntially. Hence, an intE'rmediate function, the "segment-length-function" SLF is defined in the following way:

This definition is valid for any of the paramE'ters P. It is extended to the binary features in a way that any change of these features is regarded a major one. 1. is the length of the minimal seg­ ment which efid~ at point n, whereas 1 k stands for the length of the minimal segment be~inning at n. Thus, the SLF describes beginning and end of every significant change in the controlling para­ meter and, thus, locates all dynamic segments. In eq. 6, the SLF is defined for one parametE'r at a time only. If more parameters have to be regarded, a combined SI.F is defined as:

In this equation,SLF p is the SLF as defined for n, . a single parameter P. a~cording to eq. 6. The com­ bined SLF is comput~d as the minimum of the SLF values for the individual parameters P .· This de­ finition again emphasizes the dynamlc seements, since the value of the SLF is determined by a sin­ gle major change of one of the parameters already.

After computing the combined SLF of the whole sig­ nal, the minimal segments are determined in a way that first all microsegments with SLF·I (microseg­ ments) are combined to minimal segments,after that the microsegments with SFL·2 etc. (Fig. 4 and 6).

Before that, the SLF has been smoothed in a way that the difference between adjacent SLF values cannot be greater than one. In order to maintain the exact localization of the dynamic segments, the SLF must not increase its value at any point during the smoothing procedure. This step from the microsegment to the SLF resp. to the minimal segment provides minimum dependence of the further segmentation steps on the parameters applied, and - by means of the thresholds q - also minimum dependence on the individual spea~er. Since it can influence the magnitude of the SLF but not its structure, especially the situation of its ex­ tremes, in an essential way, the individual ad­ justment of thE' thresholds q is not too critical (Fig. 6). p

4. Primary Segmentation.

The following steps labels all the minimal seg­ ments "dynamic", "stationary", or "undefined". Ad­ jacent minimal segments with equal labeling are grouped together to "primary" segments. A minimal segment certainly can be labeled dynamic when, during its course, there is a major change between adjacent microsegments. A minimal segment certain­ ly is stationary when there is no major change in any parameter durin~ a wider neighbourhood (30 ms or more). The rest of the minimal segments not he­ longing to one of these categories is labeled "un­ defined" (Fig. 6).

5. From the Dynamic to the Transitional Segment.

A dynamic section in the course of tbe speech. sig­ nal always represents the transition from one po­ sition of the vocal tract to the subsequent one. This transition can be characterized more accurately, if one succeeds to find a measure or a state­ ment for the direction of the transition, and, es­ pecially, if one succeeds to indicate whether the transition proceeds in a monotonic way or not. The really transitional segment, that means the direct trans1t1on from a spoken phoneme to the next one, should reveal a monotonically increasing or de­ creasing course of the parameters. In some pho­ nemes, such ~s stops, however, the target position of the vocal tract is not sustained; thus, a sta­ tionary segment cannot be expected at that point. In this case, a sequence of adjacent trans1t1ons is grouped together to one dynamic segment by the primary segmentation algorithm. To find a measure for the degree of monotony in a transition, the auxiliary values where L represents the number of microsegments in the m-th primary segment, are determined for all segments and all parameters. bm is a criterion for the over-all change of each parameter within a seg­ ment, whereas Cm is a criterion for the direction of change as well as for its degree of monotony. If the course of the controlling parameter P with­ in a segment is monotonically increasing or de­ creasing,the absolute value of <in will be situated in the vicinity of I. Otherwise, the value em will be situated around O.After computing these values, the dynamic segments are transformed into "transi­ tional" segments by determining their direction characteristics. Monotonic segments are labeled "transitional-rising" resp."transitional-falling". Each non-monotonic dynamic segment is subdivided into monotonic fractions if it exceeds a certain length (see Fig. 7). In order to create a unique determination of direction characteristics when several segmentation parameters are involved, in each dynamic segment, one of the parameters is se­ lected and labeled dominant for the direction char­ acteristic of that particular segment resp. the transitional segment(s) which will result from it /Hess, 1972/.

-------

6. Undefined Segments.

After the primary segmentation, about 307. nf the speech signal remain as undefined segments. These segments have been generated at those points of the speech signal where a clt>ar decision as to sta­ tionary or dynamic could not he made inunediately . A special algorithm eliminates these undefined seg­ ments. For this, it needs the classification of their neighbours. Four main categories of combina­ tions with undefined segments may occur:

1. "transitional" - "undefined" - "transitional"
2. "stationary" - "undefined" - "stationary"
3. "transitional" - "undefined" - "stationary" and vice versa
4. long undefined segments ln any environment.

The performance of the algorithm for the "short" undefined segments (categories a to c) with regard to the SLF and the direction characteristics of the undefined segment itself and its neighbours is shown in Fig.B. Two configurations may Bive ambig­ uous results: undefined segments between station­ ary segmt>nts as well as undefined segments between transitional ones,the latter only if the airection characteristics are equal in all three segments involved.In this case the undefined segment is la­ beled different from its neighbours if the change rate b in the undefined st>gment and in the neigh­ bours ~iffer strongly. If there is only a slight difference, the segments are grouped together. If the decision is not clear, the undefined segment is labeled "may-be-transitional" or "may-be-sta­ tionary" thus revealing that the result at this point is subject to some unsafety.

As can he seen in Fig. 7, it is not provided by the algorithm t o split up an undefined segment in­ to more than two segments. This may prove suffi­ cient for isolated words; in connected speech, however, there exist longer undefined segments which may consi s t of three and more phonemes. These segments are treated in a recursive way. They are assumed to contain at least one segment t o be labeled "stationary" and one to be labeled "transitional". To loca te these segments, the SLF thresholds for labeling a segment stationary resp. dynamic (see Fig. 6) are decreased resp. increased until at least one stationary and dynamic segment is found. These subsegments then are separated from the rest which keeps its undefined labeling and can be reprocessed according to one of th~ four categories.

After performing this procedure, no undefined seg­ ments will remain in the course of the signal.

7. Corrections.

7.1 Parameter F.rrors.

Since the segment<ltion parameters, for better ac­ curacy, are measured pitch-synchronously, one must regard that the speaker may utter irregular sig­ nals or that pitch detection may fail temporarily /llt·ss , 1974.3/. The effects of this type of error are reduced by smoothing the segmentation parame­ ters where such an error is detected. Usually, a slight paramt>tt>r smoothing is done where pitch is re~ular; a medium smoothing is performed on voice­ less sections. Irregular sections have to be smoothed in a way that parameter extremes due to incorrect measurements are reduced to almost zero.

Thus, if an obviously nonsense segment combination is detected which could t.ave been caused by a pa­ rameter error, the involved part of the signal pa­ rameters will be smoothed as if it were irregular, and, after that, will be reprocessed by the seg­ mentation algorithm.Among the segmentation result<; which will cause smoothing and reprocessing are the following combinations:

1. Clusters of more than three transitional segments
2. Failure of the algorithm to divide a non-mo­ notonic dynamic segment into monotonic transitional ones
3. Clustering of "may-be"-segments or very short stationary segments.

7.2 Hidden stationary segments.

Two important special cases have been remaining (Fig. 9). They represent the only errors intro­ duced by the strong emphasis of the dynamic seg­ ments in the definition of the SLF and the minimal segments:

1. A local minimum of QS within a transitional segment, especially after a pause or a voiceless segment~ points to a short glide or nasal which could not he detected due to a substantial increase in both levels L and L . At this point, the algorithm S inser~s a "may-be-stationary" segment.
2. The sequence "transitional-rising"/"transi­ tional-falling" in stationary environment or inunediately after a pause leads to a vowel being too short to form a stationary segment (e.g. a reduced vowel). Here, the algorithm inserts the missing stationary segment in the middle, if the two transitional segments are long enough.

7 . 3 Sensitivity of the SLF Algorithm with Respect to the Values of the Parameters.

There is one error associated with eqs . 4 and 5, that means with the definition of the SLF. If the value of the correction factor in eq. 4 is high, r.k represents a value about proportional to the a~solute parameter change. If the value of K is situated near zero, r.k represents the relative parameter change. In Soth cases, the results show that the values of ri~ (and, subsequently, the SLF values) associated w1tn a normal phoneme boundary depend on the absolute value of the signal level at that point. Therefore,besides the threshold q , the correction f ac tor K represents an arlditionRl degree of freedom which may balance the response of the system to a phoneme boundary with respect to the values of the parameters at the point regarded.

8. Results

As an example, Fig. 10 shows the segmentation of the German word "Be lob igung". For each of the t~r~e parameters LS, L 0 , and OS' ~he individua l m1n1mal segment sequence together w1th the course of the individual parameter are depicted; besides that, the figure shows the combined SLF, the com­ bined sequence of minimal segments as well as the final segment sequence. From this figure, the in­ fluence of the three parameters also can be seen. The signal level LS' that means the absolute aver­ age of the unprocessed speech signal, shows the

TRA Transitional

STA Stationary

DYN Dynamic

MST "May-be"-Stationary

+,- Direction characteristics

Results BEFORE AFTER Correction

greatest changes and thus is most suitable for lo­ cating dynamic segments at stops and transitions before and after pauses and voiceless sounds. Transitions between vowels and semivowels, nasals or glides,however,are better detected by the level LD of the differenced signal. The level ratio q 5 , at last, is most sensitive to changes in the over­ all frequency behaviour of the signal; for examp­ le, at the burst of the phoneme /g/.

• I Minimal segment sequence; Lg only
• 2 Minimal segment sequence; only LD
• 3 Minimal segment sequence; Qs only
• 4 Course of SLF; all parameters
• 5 Minimal segment sequence
• 6 Final segmentation

S Stationary

T Transitional

• I Voiceless
• 2 Transitional, rising
• 3 Transitional, falling

PS, PAUSE Pause

M "May-be"-segment

The segmentation procedure was applied to speech signals consisting of a list of 200 isolated Ger­ man words, each uttered by seven speakers.It turn­ ed out that, from these utterances, the algorithm missed 1.3% of the phoneme boundaries and 0.9% of the stationary segments (especially short glides). The voiceless/voiced discrimination contributed 0.3% to the . total error score,and the pause/speech discrimination participated with 0.8%. A per­ centage of 2.1% of the boundaries were inserted additionally thus dividing one phoneme into more than one stationary segment.

• Fig. II shows the performance of the segmentation procedure for a whole sentence.

When reviewing the results, one has to check whether resp. to what extent the unique relation between phoneme and stationary or transitional segment ar1s1ng from the segment definition and the phonetic content leading to a "reference seg­ mentation" is realized by the speaker at all. For example, several allophones had to be admitted for the phoneme /r/ and for the diphthongs. The error rates given above were measured in comparison with the results of a hand segmentation of the actual signals. With respect to this, the resuits seem prom1s1ng, and what limits the value of this (and almost any) segmentation procedure is not so much the inaccuracy of the method or of the algorithm, but inaccuracies introduced by the speakers, such as additional spoken phonemes, phoneme reduction, and erroneous pronunciation.

Acknowledgement.

The author thanks Dr.-Ing.R.Schrag and Dr.-Ing.H. Seidel for many fruitful discussions, and Mrs. I. Kunert for preparing the manuscript. This work was partially supported by Deutsche Forschungsgemein­ schaft (Germen Research Association) within SFB 50 Kybernetik (Special Research group SO: Cyberne­ tics). Parts of the algorithm were run on the TR 440 computer of the URF Informatik (National Re­ search Program in Computer Science).

References.

• /Bhimani,l963/ B.V.Bhimani: Multidimensional Model for Speech Recognition. Def.Doc.Center Alexandria (Va.), USA, 1963.
• /Denes,l969/ P.B.Denes, T. von Keller:Articulatorv Segmentation for Automatic recognition of Speech. Proc.bth ICA,Tokyo 1968: American Elsevier Publ.Comp. 1969.
• /Delattre,l968/ P.Delattre: From acoustic cues to distinctivP features. Phonetica 18 (1968), pp. 198 ... 230
• /Fant,l961/ G.Fant and B.Lindblom: Studies of Min­ imal Speech Sound Units. STL-QPSR 1961 Nr. 2, pp.I ··· IO
• /Flanagan,l972/ J.L.Flanagan: Speech Analysis, Synthesis, and Perception. Springer-Ver­ lag, Berlin, Heidelberg, New York 1q72 (2nd ed.)
• /Hess,l972/ W.Hess: Digitale grundfrequenzsynchro­ ne Analyse von Sprachsignalen als Teil eines automatischen Spracherkennungssy­ stems. Dr . -Ing. thesis, Munich 1972.
• /Hess,l973/ W.Hess: Time-domain segmentat i on of speech signals. Contribution to a work­ shop on segmentation and labeling of speech signals, held at Carnegie-Mellon University, Pittsburgh, Pa., USA, in Jul y 1973 (proceedings to appear . in 1975).
• /Hess,l974.1/ W.Hess: A pitch-synchronous, digit a l feature extraction system for phonemic recognition of speech. In : Proceedings of the IEEE symposium on Speech Recognition, Carnegie-Mellon University, Pittsburgh, Pa, USA, 15-19 April 1974. Ed. by L.Erman. IEEE Press, New York 1974 .
• /Hess,1974.2/ W.Hess: A pitch-synchronous, digital feature extraction system for phonemic recognition of speech. In: Proc.eedings of the Speech Communication Seminar (SCS) Stockholm, Aug. 1-3, 1974. Stockholm 1974 : Almqvist and Viksell.
• /Hess,l974.3/ W.Hess: On-line, digital pitch per iod extractor for' speech signals. In: Proceed­ ings of the 1974 International Zurich Sem­ inar on Digital Communicat i ons, paper AS . 1974: Zurich; Switzerland.
• /Hughes,l965/ G.W.Hughes, J.F.Hemdal: Speech Anal­ ysis. Purdue Res. Found. Techn. Rept . TREE 65-9, 1965.
• /Lea,l972/ W.A.Lea: An Approach to Syntactic Recog­ nition without phonemics. In: Conference Record, 1972 International Conference on Speech Comm. and Processing. AFCRL, Bed­ ford, Mass., USA 1972.
• /Olson,l967/ H.F.Olson et al.: Speech Processing Techniques and Applications . IEEE Trsa. AU-15 (1967), pp. 120 .·. 126
• /tlhman,l962/ S.E.G.tihman: Perceptual Segments and rate of change of spectrum in connected speech. Proceedings of the Speech Communi ­ cation Seminar, Stockholm 1962.
• /Otten,l964/ K.W.Otten: Segmentation of continuous speech into phonemes. Rept.Nr.RTD-TDR-634005, Part 2, U.S.Army, 1964 .
• /Paulus,1974/ E.Paulus, R.Schrag, and Th. Scho t o l a: Advances towards a four-stage system f or automatic speech recognition. In : Proc. of the Speech Communication Seminar, St ock­ holm 1974.
• /Pols,l969/ Pols,L.C.W., J.T. van der Kamp, and R. Plomp: Perceptual and Physical Space of Vowel sounds. J.A.S.A. 46(1969), pp. 458 ··· 467
• /Reddy,l966.1/ D.R.Reddy: Segmentation of speech sounds. J.A.S.A. 40 (1966), pp. 307 · .. 31 2
• /Reddy,l966.2/ D.R.Reddy: Phoneme Grouping for Speech Recognition. J.A.S.A. 41 (1967), pp . 1295 .·. 1300
• /Reddy and Vicens,l968/ D.R.Reddy and P.Vicens: A Procedure for the Segmentation of con­ nected speech. J.Audio Eng.Soc. 16(1968) , pp. 404 ... 4 I I .