Microphonemes As Fundamental Segments Of Speech Wave Primary Segmentation - Automatic Searching For Microphonemes

Introduction

System AKC!I ID was designed for research work concerning the analysis and synthesis of speech waves by means of a ZAJ'-1-41 com­ puter ( 1, 2, 3 ) · Durine the initial phase of r e :>earch that system was helpful in defining the most essential problems of speech wave analysis. It also made it pos­ sible to inculcate and test some algorithm models of speech analysis. However, before accepting a concrete algorithm model of speech analysis it was necessary to re­ alize different functions of the hearing organ. Those functions are expressed by an analog model of the human ear (4). Generally, the traclc of a given sound in the ear may be presented as follows: the external ear, the ear-drum, the internal ear and its bones (malleus, incus, stir­ rup). The movements of the stirrup bone cause translocation of cochlea septum : -its resilience is not constant through its whole lenGht (4,5). That vibration is subsequently trans·ni tted through the nerve cells on the septum to "'30 000 auditory nerve fibrils. Figure 1 shows analog model of human ear presented by P.Kolers ( 4) · Althou~h this model is only approximate,a comparison of the characteristics obtained experimentally showed a considerable con­ vergence (4). de accept the conclusion, with no deeper consideration of the ear model presented here, that the internal ear is the centre of sound spectral analy­ sis 3(t) · Tts t'esults are made avera3e nerves PCt) -external auditory duct pressure, G(f) -linear block transforming pressure P(t)into voluminal translocation s(t}of the stirrup, H(f,x) - this function con­ nects translocation s(t)with translocation of septum d(t,x)at the point x[cm]from the stirrup, rms - realises the mean-square function of input signal d(t,x)for short duration ("' 10 ms).

Figure 1

in time for the period Ts C~10ms).The mo­ del of the auditory organ presented here is a basis in this paper, although its im­ perfection is realised. But it is not a novelty. The majority of studies concern­ ing the analysis of speech waves have been based on the spectral analysis of the exa­ mined signals (6). Some of them used the dynamic spectral analysis, yet they were strictly based on accepting constant in­ tervals of 10-20 ms. Although .the accep­ tance of t hat constant interval made it possible to realise some functions of ana­ lysis more simply, it caused some inaccu­ racy at the level of parametrization due to a considerable dispersion of parameters of particular segments(12). This fact re­ sulted in substantial difficulties con­ nected with the segment extraction for re­ cognition. It seems relatively easy to find a relevant set of distinctive phoneme features and so the segment corresponding with phonemes would be the most suitable for the recognition (8). The number of segments determining particular classes also points to the same conclusion; in the case of phonemes there would be no more than 40. Yet, difficulties connected with phoneme segmentation as well as those resulting from the lack of univocal depen­ dence between the parameters of some con­ crete realization of the stochastic pro­ cess and the process itself, make it too difficult to work out a highly efficient automatic system of speech recognition. 3ome authors consider it simply impossible to develop studies bas·~d on phoneme analy­ sis and recognition (7,8) pointing to t he practical difficulties of segment extrac­ tion. 1t is po:>sible to c :>nclude t '1at a solution of this problem will becone a ba­ sis f or the development of some nethods of speech analysis by :neans of particular speech sounds of a given natural laneuage.

The process of segmentation \ihen properly conducted will help in solvin6 some other problems of speech wave analysis e.g. time compression, coding, code ciphering and re­ cognition. f'lany scientists have noted the role of segmentation in the process of speech wave analysis ( 6,7,8,9) · Yet, they were too strict in their choice of segments rterived f rom the analog model. Naturally, therefore, they did not obtain the type of results which could be acquired through a segmentation based on the segments chosen in the most optimum manner. This paper at­ tempts to indicate the algorithm of auto­ matic segment~tion based on microphonemes as the dynamic segments.

f'licrophoneme as the Fundamental Segment of Voiced Speech ,1laves

We are concerned with voiced speech 1 since only for that class of speech is possible to extract fundamental segments in a simple way, keeping in mind the purpose this ex­ traction could serve. It is also possible to resolve the stochastic signal into the class of stationary signals (as considered in frequency domain) by means of some sim­ ple methods, for examples, Fourier's trans­ formation. It corresponds with the model of the ear presented above. If we consider the objective function in speech wave ana­ lysis the following requirements arise:

• for the purposes of information reco­ gnition (its content) it is necessary to accept segment Sk, L = 1, 2, 3, ··· , maldng it possible to realize the set of relations m< for the correct classification, while con­ sidering the subset of parameters iR = t defined for a given segment. It may be presented as the following condition:
• 1 mi. : f~<a(S~) c · fRi. ; Ls 1,2, . .. meaning that there is a relation miwhich makes it possible to transfer the set of parameters defined for any "l" realization of the class Ki, t =1,2, ···· ,m ("l" segment S,.) into the set of parameters f~ti. relevant to class Kt ·
• for the purposes of compressing and coding a speech wave, it is necessary for the extracted segments S~ to satisfy con­ dition No 1. Considering the time compres­ sion, the best solution (if possible)would be to extract such segment SK so that they will be realization of the quasi-periodic waveforms within a given class Kt · That quasi-periodicity should be conceived as the periodicity which is discrete from the point of view of segments (time quanta of real signals) · In other words the segment SK would constitute a string of the seg­ ments 'SIC

In the time domain ! (continuous) it would be possible to present it in the following way :

L·1 TaO where:SK means the segment SK defined for a period of time tK (conceived as a set of samples of a signal in the interval of[o,tJ), n. is a function of time t and it is cal cu­ lated according to the follo'.ting condition

It is ill~strated in Fi gure 2.

Condition No 1 determines the time for the Segment 51( ; it iS fK ·

Considering the function of compression and coding,tK should be the maximum length of time of the segment SK satisfying . con­ dition No 1. Fundamental segments ~~ forming quasi-periodicity of the segment SK will be called primary segments. Each of them is of different duration T(j) and they form quasi-periodicity within the se­ gment S1< : - see formula No 3. ':lith regard to the function of speech wave time com­ pression, primary segments should satisfy the following condition :

This condition states that a number of

primary segments n . should be maximum. - for the purposes of speaker identifi­ cation it is necessary to indicate such segment s~ ; l =1 ,2,3, ··· ' for the subset of parameters afterwards defined (iden­ tifying the speaker) to satisfy condition No 1 in consideration of a given class of speakers K p and also

This condition indicates that the subset frp of parameters identifying the speaker specified in consideration of a freely ac­ cepted segment Sz (free in the sense of time ) is thp equivalent of the subset of parameters trp · These parameters are de­ fined by means of one of the possible methods and the subset is calculated with some optional duration of speech for a given speaker, belonging to th·) class Kp · - in the process of reducing the noise the most essential parameters are : the speech spectrum and particular]fthe ave­ rage spectrum of a speech wave. Let us de­ note the subset of those parameters : f 5 · \·/i th this approach it is necessary to de­ fine the segment So such as :

This condition means that it is necessary to indicate such a segment So for the calculated parameters of the average speech spectrum, to f ix univocally the average spectrum parameters calculated for a speech wave of any duration and at any moment. In this case it is necessary to in­ dicate a segment which would satisfy condi­ tion No 7 and which would at the same time be the most favourable considering the mi­ nimal time interval. This approach shortens the process of indicating the parameterslc considerably. ':11th regard to these conside­ rations, it is necessary to state that in the process of speaker identification and noise reduction, the choise of the best segment should be carried out considering its length of time. The shorter the segment S: or So , the larger the possibilities of realizing both processes. Therefore, it will be possible to,indicate the minimal segment of a speech wave and through that to identify the speaker. The segment chosen in this way must satisfy condition No 6 and 7, corresponding with condition No 1 in that the set of parameters is indicated for a given speaker p · But for the purposes of corJ.pression and codification it is neces­ sary for the segment SK to be maximum and also to satisfy condition No 1 as in the case of recognition. In this c~e the aim is also to find thP. segment SK j> which would satisfy conditions No 1,3 and 5. We must search for a primary segment satis­ fying the following condition :

within a given class Ki. , i. =1,2, ··· ,m. Because it is well known that the acceptan­ ce of the se~ent SR fixes the number of classes Ki. ,( i. =f(SR)), the minimation of the number of classes Kt is the additional condition that should be taken into consi­ deration while choosing the segment SR · If we accept syllables and words as the segment SR , the number of classes in which it would be necessary to include the ana­ lysed segments would be tens and hundreds of thousands. Therefore it seems that it is necessary to accept a phoneme as the se­ gment SR or a primary segment which is the equivalent of phoneme considering the con­ dition No 1. The number of classes with a segment so defined is m a 40 for the Polish language. With this number of classes a given phoneme as a segment Sq is the lon­ gest segment and it would fix the segment St< within the class K~ · Yet in spite of its usefulness for the purposes of recogni­ tion and compression, it would not be the most favourable segment because the condi­ tion 5 would not be satisfied. And only finding the primary segment that would meet all the requirements would make it possible to state the existence of the most favoura­ ble choice of the speech wave segment, in terms of recognition of the content and time compression of a speech wave ·

The number of classes according to con­ dition No 8 would be retained. The additio­ nal effect of such an approach on speech wave analysis would be the obtaining of a dynamic analysis of parameters defined within the primary segment "S>< · Concerning the two remaining goals, speaker identifi­ cation and reduction of noise, we may ac­ cept the primary segments 'SK as fundamental only if it is possible to determine the parameters lr-, and Ta"· on the basis of any parameter combination estimated within a given seement ~~< · In the case of speal<er identification is advisable to estimate parameters for the dynamically analysed speech wave. Sp~ed and changes i~ the larynx vibrations are some of the dynamic parameters characterizing the speaker and his articulation : let us denote the len~­ th of time of the larynx tone : T~ · It is a function of time for a given voi­ ced speech wave. Therefore it is advisable to accept the primary segment~><; its du­ ration is r {j) :a T K · It is different from the majority of algor! thms assuming T(j)= const · This approach would also give the phase accordance of the analysed seg­ ments (11). The primary segment is based on pitch period equal to TK corresponding to the analog ear model (T~ ~ 10 ms) and is called a microphoneme in this paper. Our studies have entirely confirmed the pertinence of the accepted reasoning and also the fact that a microphoneme satis­ fies all the above conditions (12,13).

Prim~ Sesrentation Automatic Sear~ing or Mlcrophonemes

Since a voiced speech wave results from the vibration of the vocal cords and is therefore only approximately periodic and since it depends on the individual characteristics of the speaker, it is not possible to assign definitely the frequen­ cy which could be considered fundamental for all segments S of a speech wave. In other words, the length of time of micro­ phonemes is: T(j) ~ const. The purpose of primary segmentation is the extraction of microphoneme sequence from the continuous speech wave of a given speaker by means of an algorithm realized as a computer pro­ gramme. The solution of the problem con­ cerning the extraction of quasi-periodic segments of a speech wave is complicated by the fact that not only quasi-periodici­ ty is troublesome but also because the speech waves vary in amplitude and shape. In assigning microphonemes the algorithm uses some of the methods and remarks in­ cluded in Reddy's algorithm : Pitch Period Determination (10). It also completes Reddy's algorithm with the elements con­ nected with the acceptance of the micro­ phonemes as a primary segment and with other parameters appropriate for Polish speech as well as the accepted method of their representation ( 1 ) · The algor! thm PPD was used as an element to indicate the location neighbourhood of the expected end of a given microphoneme. The algorithm extracts microphonemes and indicates t he places of expected microphoneme ends of the analysed speech wave and also gives a proportional indication of the voiced sig­ nal content in the analysed speech wave.

As we have already assumed, we shall be searching for the microphonemes only in that part of the speech wave which is vo i ­ ced according to condition No 3. Therefore the first step of the present segmentation is to find and indicate voiced segments of the analysed speech wave. Because the whole procedure of primary segmentation operates on the time signal of the speech wave, the first step makes use of amplitude and fre­ quency criteria · At this stage these cri­ teria make it possible, as well as in the PPD algorithm, to distinguish three funda­ mental kinds of signals : silence, noise and a quasi-periodic signal. The algorithm has been adapted to the processing of the input signal which is first quantized. It assumes the frequency of sampling : fp =12 kHz, which makes it possible to represent t he primary speech wave quite well. The analog-digital converter working in the AKOHD system performs signal quantization on 256 levels, enabling the notation in -the form of numerical data ranging from [-128, +127] · This fact has its reflection in the values of coefficients indicating the amp­ litude thresholds. Initially the following assumptions have been accepted as in Reddy PPD :

1. In any examined speech wave (with the same speaker) the length of neighbour­ ing microphonemes cannot differ more than 20%
2. :/hen the frequency of the larynx tone is contained within 70-450 Hz the fre­ quency test fp ""12kHz will produce a length of microphoneme within the bounds of 26-171 samples.
3. The examined speech wave will be also accessible in the possible process of correction. This assumption is easy to realize in the case of computer proces­ sing.

The succeeding speech wave segments of 0 · 256 samples ( 21,3 ms) undergo the amp­ litude criterion. The accepted length 0 results from the fact, that in the segment S· longest microphoneme ~~ should be con­ tained (max ( T ( j Hp)<D).

The segments S 1 o , l =1 ,2, ··· are accepted as silence segments if the maximum amplitude within a given segment does not exced the threshold value 0.. = 8 , o -times. In other words a variable ~'

( where Xi c. 5~ means the samples o . ~- the segment L ; its length is 0 whereas #{ } means the cardinality of the set )

assings some segments SD accepted as si­ lence in the case of the condition a:~< 6' In this algorithm o·5 was accepted. At the next stage only those segments S~ are ac­ cepted for which oc~ ~ 5 · All those seg­ ments (of t he length 0 ) having o:~L)6" \ihere

~ =43 are accepted as the segments of quasi-periodic character. All the remaining segments undergo the frequency criterion. Initially they are included among the class of high frequency signals or among the class of voiced signals or "conditional noise". Frequency criterion is based on the estimation of zero-crossing parameters. The segments So for \'lhich the number of zero-crossings is larger than 6;= 96 (and that with fp= 12kHz equals the frequency exceeding 2,5kHz ) are considered high frequency noise segments. ·.1hereas the seg­ ments for which the number of zero-cros­ sings exceeds tr., = 50 ( 1, 2kHz ) and their maximum amplitude is max { .Xl}<6'2 ; x 4c: S~ are accepted as "conditional noise".

"Conditional. noise" may be accepted as a voiced segment or as high frequency noise. If there is a group of "conditional noise" segments in the close neighbourhood of the segment accepted as high frequency noise, the whole group is accepted as high fre­ quency noise. otherwise the "conditional noise" segments are considered and inclu­ ded among the class of voiced segments. If we denote the segment ~accepted as si­ lence - 0, voiced segment- 1, 11 conditio­ nal noise 11 - 2, and high frequency noise - 3, the signal represented by the follo­ wing sequence of the approprite classes of the segments So

000111000111222111 2233112233111

is classified as

0001110001111111113333113333111 During the next stage of the algorithm we make use of the results of the previous stage. Only those segments which have been accepted as voiced are exposed to proces­ sing. It is necessary to search for inde­ xes i for which the momentary values cor­ respond with the significant amplitude of a given microphonemes. As well as PPD al­ gorithm the following terms are introduced - local maximum and local minimum, absolute maximum, significant maximum and minimum and also significant extreme. Some modifi­ cations concerning the semantic content of the above terms have been introduced and also the algorithm of their indication has changed.

If we denote the vector representing the input signal as X then:

X< E )( is a local maximum if (Xt > Xt-t)/\ (Xt > Xiot )A ( Xt -:)0) and xt E X is a local minimum if

(Xi.<Xi-t)A(Xt< Xt-.t) 1\ (Xi<: 0) 'de shall denote them maxl.x, minl x respe­ ctively. On the basis of local maxima the absolute maximum, the significant maxima and significant minima are indicated. ':le denote them max.z.,.x, minz,X, A. The absolute maximum A is calculated for the succeeding segments of the length 01 = 80 ( it makes about 170.6 ms ). Here, it is necessary to point out that a division of a signal into segments of the length 01 as dictated by the limitations posed by the ZAH-41 is different from the pri~ciples given in the algorithm PPD and aims at using as fe\i me­ mory cells as possible and achieving co:n­ patibili ty with the system AKORD ( 1, 3 ) · Accepting the segments of the length ::l1 greatly inffluences the process of dyna:nic analysis of the microphoneme periods. It is also extremely significant in the pro­ cess of correction, during the final stage of algorithm. ·rhe indication of t he period of the larynx tone for such segments al­ lo\'ls in::iependence f rom f airly essential changes of the length of the larynx tone owing to some changes in articulation, contrary to the indication of the expected period of the larynx tone estimated conti­ nually through the w'aole signal X.

It is particularly important for long sig­ nals considering the initially accepted assumption No 1. ·rhe indication of segmen­ ts of duration D1 influences the first sta­ c;e of the algorithm in which 3 segments 3 · are indicated successively for each step · . H thin every segment S~ 1 , k =1, 2, · · · the absolute maxi~um Ak is assigned.

where {maXtnX}k <Jeans the set of loc~l ma­ xima indicated within the segment SD 1 · rraving Ak , we indicate the significant maxima and minima within the sets of local maxima and local minima. The significant maximum must satisfy the following condi­ tions:

where p{max1nx,ma'1Zn-tx) indicates the dis­ tance between the previous and the current significant maximum. That distance should be minimum · ..,rithin the defined i:!'lterval. If the condition :11 is not satisfied the first "essential" local maximum is chosen according to the condition:

''12 ' · mcu_a,X) u.nm{lfkl.\X =~-'.s>(mcu,r,llllllat-tX)\�J }A A �2,f(~X.mcu1,..1X)~�3"j(mat1,.x,mcu1Jt'1r)::rmln 9

If Lhe sl ;nificant maximum has been indi­ cated considerinG t he condition 'l/1 and if that condition is satisfied:

p (maxZnx,maXz:n-lx)>2·62

then we make an attempt at finding the additional significant maximum maxtnx according to the condition ':12 wi th~n the interval satisfying the following inequa­ lity:

6'2~,9(max_T!x,malCz~){~(max1nx,max1". 1x) -Gz This algorithm is different fro~ the one presented by R.eddy, and thus t he signifi­ cant extremes will have different meanings than in the algorithm PPD. Yet o·t~ing to this difference,it is possible to elimina­ te errors of the type of amplitude fluctu­ ation caused by lo\-1 frequency ( T a 2Tk) · The algorithm of searching for the signi­ ficant minimum i s analogous to t he search for the significant maximum. .Speech wave analysis has shown that because the enve­ lope of the larynx tone is of a rather differentiated character, not all signifi­ cant maxima univocally assign the places of extremes of the microphonemic envelope. It can be observed that whenever there exists a significant maximum tha ~ we ac­ cept as the extreme of the microphoneme envelope, there also exists a correspond­ ing significant minimum wit!'l.in a small neighbourhood. This can be explained by the occurence of the ~reatest turbulence in the speech wave a t the moment when the vocal tract is excited by the release of the pressure resulting from the vibration of the vocal cords, the moment in time when the microphoneme begins · Therefore, as the "significant extremes" only those significant maxima are chosen which have the corresponding si~ificant minim~ in their neighbourhood (4.5 ms- 42 samples as 1/4 of the longest expected microphone­ me) · Thus obtained, microphoneme markers undergo the logical processing. During t he stage of the logical processing we make use of the already presented assumption that microphonemes are searched for in the quasi-periodic si~al and that the follo­ winr, inequality is satisfied:

This assumption makes it possible to eli­ minate t~e errors of the second stage, e. g . finding an extreme in a place where it should not be or not finding it in the pla­ ce where it should be. In order to elimi­ nate such errors an expected value on the average period of microphonemes Tsp is cal­ culat ed. It is calculated as a discrete statistical distribution of distance among t~ selected extremes within the segment s~ , consistent with the earlier ac~p­ ted settlements. '. le shall denote it Tsp · Thus calculated, T,J'p is used for the stage of correction on the same analogous bas is as it was done by Reddy (1 0) · The accep­ tance of T~ calculated along the fi xed length Dt of the segment So 1 makes it pos­ sible to eliminate the errors, resul ting from intonation changes of longer state­ ments. Thus the correction's efficiency has been considerably increased while re­ taining all of Reddy's algorithm and the coefficients accepted by him.

The process described above concerni ng the logical processing is iterated for the same segments according to need. The re­ trieved markers indicate the significant places within the microphoneme and those regions make a basis for finding the ends of microphonemes. The author will cons i der the zero-crossing point or the point vmere the signal assumes a value which is t he closest to zero, the end or begining of the microphoneme. The end or begining of microphoneme is indicated in the region where the envelope of the signal assumes a minimum. Such a choice for the place of the microphoneme end reflects the phys ical aspect connected with the excitement of t he vocal cords and also it corresponds wi t !1 the results of the spectral analysis. The spectral analysis concerning microphonemes showed great parameter stability. 1'he agreement of phase as well as the great relevance of microphoneme and phoneme cha­ racteristics, belonging to the same speake r and to the same class of speech sounds, proves the correct segmentation. In accor­ dance with the phenomenon of vocal cords vibration we shall search for the place of the microphoneme end (or the be "; ining) before the maximum of turbulence assigned by the s i gnificant extremes. ·re :search for t he ends of the ~icrophoneme before t he s ignificant extremes or before the pro­ ceeding local maxima. ':le accept a princi­ ple that the end of the microphoneme is calculated directly before the local ma­ ximum which is the furthest on the left

and its value is not less than 1/2 of the value of the succeeding significant extre­ me. Ict there is a condition where the di­ stance between the indicated end of the :nicrophoneme and Lhe succeeding extreme should not be larGer than.0.5 Tsp · Let us denote this distance as E"' ; where j is a succeeding number of the microphonemc. It is used for indicating the average distan­ ce bet'.'lecn the expected end of the micro­ phoneme and the succeeding extreme.

This value is used for the correction of erroneously calculated ends of micropho­ nemes. Such errors may appear in the situ­ ation shown in Figure 3.

All the ends of microphonemes which satis­ fy the condition:

W 3 ·

IEsA-Ejl \ 7i-Tsp

are considered correct.

In the other cases, a process of cor­ rection is undertaken which may lead to indicating a new and better end of the mi­ crophoneme. The al~orithm of correction aims at finding a new end of the micropho­ neme, satisfying condition W3 · This pha­ se ends the stage of correction for a gi­ ven m:l.crophoneme or the stage of indica­ tion of the microphoneme end, which is the least distant from the expected end, mar­ ked by f~,SR · The ends of microphonemes are calculated durin~ the stage of correction according to the analoeous principle as before but the values of succeeding local maxima need not exceed half of the value of the respective extreme. Because the process of correction may be repeated, there is a condition indicated, concerning t;1e character of this process. The process of correction should be "unilaterally convergent", by which we mean that a new end of the micronhoneme will be considered correct, and the-stage of correction in which it was calculated will be considered essential, if it is placed closer to the expected end than the previous incorrect one and als0, if it is on the same side, in relation to the expected end, as t·1e end considered incorrect. In the case of 11 oscillation 11 of the correction proccess towarus the expected end, \'le accept the end of the microphonerne indicated durin[; the previous stage of correction as cor­ rect. 1\.s a consequence of a possible cor­ rection we obtain the markers of microphonemes considered the results of the algo­ rithm of primary se0mentation.

.3uuming up, the algorithm of primary segmentation is divided into four stages:

:.> ta3e I - The indication of a 'l.Uasi-~crio­ dic signal and its e;<traction

from an input si~nal.

Stage II- l'he calculation of the signifi­ cant extremes of a voiced signal.

Stage III - The correction of the extremes calculated in stage II

Stage rr - The calculation of the ends of microphonemes

part I An indication ::Jf the expected ends

part II - The correction : an indica­ tion of the resultant ends of microphonemes.

Conclusions

The procedure described in this paper was used for the analysis of 10 single words~ v1i th satisfactory results. In the majority of cases this algorithm found more micro­ phonemes than a man could have done on his own. In particular this cases concerned word endings. It was impossible to carry on more exacting research on longer signals because of the ZAM-41 computer's slmmess. Due to this difficulty, the above descri­ bed algorithm is processed in Fortran on the H-6030 computer. This new pro:;ramme will thus make it possible to test a lar­ ger number of longer utterances. Therefore, by working with more and longer utterances, it will be possible to indicate if this algorithm is better, in any cases, than those previously used for frequency analy­ sis of speech sounds.