Feature Extraction And Sentence Recognition Algorithm In Speech Input System

Introduction

In this paper two major components of the speech understanding system are discussed. One is a feature extraction method for the speech wave and the other is a sentence recognition algorithm. Many speech pattern recognition systems have been made to classify spoken words and a few have been tried to treat spoken sentences. A speech recog­ nition system which employs information from all levels - from the acoustic to r:1e semantic - to understand the lllCaning of r:-.~ .;;:;c;ech is of current interest 1 .-~ It is true that the reseaches at the higher levels such as the syntactic and semantic levels have not been sufficiently applied to speech recognition 1 : However, the balance of each level of the system is important. The application of the information at the higher levels might improve the total performance of the system but it should be noted that the communication system in which little redundancy is remained is deprived of vari­ ability and adaptability. Though the information at the upper level may be used in the manner of human speech understanding, it does not mean that he cannot correctly recognize a single word which is pronounced clearly. Therefore, the more effort should be necessary for the recognition of words or phonemes, if the speech understanding system is objected. In this study, first, a feature ex­ traction method for the speech wave is treated as the most elemental problem.

Recently, several authors have studied the esti~~tion of the vocal tract shape from the speech wave. But till now the results are not necessa­ rily satisfactory, because, as is well known, the spectral characteristics of the speech wave do not correspond one to one to the vocal tract shape as an acoustic tube and because the speech wave carries the effects of the vocal cord oscillation, noise from the turbulence and so on. Then it is desirable to apply the knowledge from physiological and phonological study for the estimation of the vocal tract shape. And such a reseach has potenti­ ality to make possible the estimation of the motor commands that move the articulatory mechanlsm. It is doubtful that, if the motor commands are precisely estimated, the phoneme recognition

FEATURE EXTRACTION AND SENTENCE RECOGNITION ALGORITHM IN SPEECH INPUT SYSTEM

K. Shirai

Department of Electrical Engineering, Waseda University, Tokyo, Japan

becomes easy. However, they may be good features for the speech recognition. In this study, an articulatory model is constructed on the basis of X-ray data and a nonlinear regression method is used to estimate the articulatory parameters. The results of the estimation preserve the typical nature of each phoneme and the method would pro­ vide a useful feature extraction technique.

Second, the problem of sentence recognition is mathematically formulated as an optimization problem with the constraint of sentence structures and is solved by a method of dynamic programming.

Here, it might be necessary to define what is recognition of sentence. The fundamental situation of the speech is conversation. In the conversation between A and B, the representation " B understands the sentence spoken by A," has too much content. Then we consider only," B re­ sponds to the sentence spoken by A." The response is described by a state transition of the machine. Of course, the machine may exhibit some outputs at the transition.

Each moment of the conversation is accompa­ nied by a· scene and usually the speaker and the receiver have a common recognition of the scene. The concept of the scene is naturally taken into account by the state of the machine. Probable sentences that may appear under a state are limit­ ed an' the effective number of the sentences that affects the recognition score is reduced.

In a practical application, the purpose and the ability of the .machine is always limited, and the contents of the conversation may be finite. Then, it is allowable to set sentence structures to order words in restricted ways.

In the framework mentioned above, the sentence recognition can be considered on the extension of a classification problem and an ef­ fective optimal algorithm for the sentence recognition can be obtained.

Feature Extraction via Estimation of Articulatory Parameters

Construction of Articulatory Model

An articulatory model presents an effective representation for the structures of the articu­ latory mechanism and the dynamical characteristics of the articulatory motion, and further it relates the articuatory parameters to the acoustic ones~ 9

The configulation of the articulatory model is shown in Fig.l. The midsaggital vocal tract outline can be represented by the variables specifying the positions of the movable structures, i.e. jaw, tongue and lips. The maxilla, rear­ pharyngeal wall and larynx outlines are fixed and approximated by the sequence of circular arcs and straight lines. The jaw is assumed to rotate with the fixed radius FjJ about the fixed point F j, and its location J l.s given by the angle Gj with respect to the reference line which is tangent to the hard palate. The jaw movement executes the passive effect to the position of the tongue and lips, and influences not onl!' ~he mouth opening area but the overall vocal tract shape.

The lip shape is specified by the height Lh and the protrusion Lp relative to the jaw position on the midsagittal plane. Only the lip protrusion parameter may be necessary to specify the lip move­ ment for the vowels, but both are required to explain the different gestures in the usual speech containing labial consonants,

The tongue contour is described in terms of a semi-polar coordinate system defined with ref­ erence to the jaw position. The center Ft rotates synchronously with the jaw movement~ 1 Therefore, the tongue contour is measured with the jaw based coordinate system. Though the tongue may be able to form various shape, it has limited freedom to move about in articulatory process on account of the physiological and phonological constraints. Thdse constraints can be expressed by the strong correlation of the position of each segment along the tongue contour and may be extracted from the statistical analysis of the X-ray data,

It is known that for the tongue articulation of vowels, the extrinsic muscle activity is more significant than that of the intrinsic one. Then, the principal components for the extrinsic actvity are obtained from the tongue contour data for vowels, and the tongue contour vector for vowels Xv can be expressed in the linear form as,

where, Vj (j·l,2, ··· ,p) are eigenvectors and~ is a mean vector for vowels which corresponds to the neutral tongue contour. The eigenvectors are cal­ culated from the next equation.

where Cxxv is the covariance matrix which is defined by

and A is the corresponding eigenvalue to satisfy the characteristic equati.on.

For the consonants, the effect of the intrin­ sic muscle activity appears particularly in the front part of the tongue but it is difficult to separate precisely the intrinsic muscle activity from the extrinsic one. At first the contribution which comes from the extrinsic components are subtracted by projecting the tongue contour vector Xc to the vowel space which is spanned by the eigenvectors for the vowels. And the remainder lc is calculated as,

where Xc means the projection of Xc to the vowel space. Again the principal component analysis is performed on Xc and the eigenvectors Ck (k=l,2, ·.· ,q) is calculated in the same manner as Vj. Finally the expression for the consonants can be obtained as, where Xc is the sum of the mean vector Xc and Xv· Then aj (j=l,2,···,p) and bk (k·l,2,···,q) become the articulatory parameters for the tongue, The above procedure was applied to the X-ray data by J.S.Perkell~

The extracted principal components of the tongue contour and their cumulative distributions are shown in Fig.2. It is found that the four components (p=3, q=l} account for roughly 96% of the tongue data variance.

The first component represents the movement of the tongue body between the rear-pharyngeal wall and the hard palate direction and produces mainly an antisymmetric perturbation of the vocal tract. It indicates the opposite feature of back and front vowels, i.e, (a] vs [i]. The second component represents the movememt of the tongue to­ ward~ the velum and produces a symmetric perturba­ tion that is effective for the rounded vowel [u]. The third component is less clearly explained and may be interpreted as the resulting tongue deformation from the contraction of the posterior fibers of genioglosus and the intrinsic muscle of the tongue tip. The fourth component is an intrinsic component and represents the tongue tip retroflex.

As an example the loci of the tongue move­ ment on a1-a2 space for three utterances /hatV/ (V: a,i,u) are illustrated in Fig,). The points 6 on the loci indicate the onset and the offset of the tongue tip closure.

From the above discussions, it is seen that the following articulatory parameters are enough to describe the midsagittal vocal tract outline, i.e. the jaw angle 0j, the lip protrusion Lp and the weighting coefficients of the tongue compo­ nents aj, bk· For the vowels the lip height is dependent on the lip protrusion and can be approximated by,

The relation (7) was determined from the analysis of the front and side photographs.

The cross-sectional dimension along the vocal tract is determined from a semi-polar coordinate system fixed with regard to the maxilla and the rear-pharyngeal wall. The relation between the cross dimension d and the cross sectional area S is approximated by power function S·2c! t,fi, In the labial region the area is approximated by an ellipse with the width given by

where Ls is the vertical separation of the lips. The vocal tract is devided into 30 uniform cylindrical tubes and the reflection coefficients between the adjoining sections are calculated. Regarding the losses at the glottis, lips and within the tract, a transmission-line model is constructed and the transfer function is expressed using z-transform.

where z=exp( j2nf~T ), ~T=2h/c, his the length of one section and c is the velocity of sound. The formant frequencies are calculated from Eq.(9) by the Fibonacci searching method.

Estimation of Articulatory Parameters

It is well known that the vocal tract shape is not uniquely determined from the spectral characteristics of the speech wave without the additional constraints in terms of the speech production process. Such constraints must be con­ sidered from the physiological, phonological and personality points of view. The constraints can be reflected on the articulatory model in two ways. One is in the physical dimension of the articu­ latory organs and in the components vectors Vj and Ck. The other is the manner of the control of the articulatory parameters. The latter is considered in this study only a little. It is clear that the number of the articulatory parameters is much smaller than that of the cylindrical tubes to de­ scribe the vocal tract shape and this fact will make the estimation easy. Therefore, the vocal tract shape is determined by estimating these articulatory parameters. '11le present model is useful for the estimation from the speech wave, because it is constructed not only to describe the articulatory state strictly but also to be directly related to the variation of the vocal tract shape. However, there remains a possibility to bring about a freedom in the articulatory parameters for a certain region of the spectral characteristics of a speech sound. Therefore, it is desirable to apply the cooperative relation between the art:l.culators in static and dynamic senses to avoid such a freedom.

The time constants of the articulatory motion are large compared with the sound propagation phenomena and in the case of the articulation of vowels, the power spectral density of the speech sound is considered to be stationary in a small interval. Therefore, the static correspondence between the articulatory parameters and the acous­ tic ones is very important.

The relationship between the articulatory parameters and the acoustic features is formulated in nonlinear regression as,

where yi is an articulatory parameter, f. is an acoustic feature and P 0 , Pii and Qi'k arJ the regression coefficients. Since Eq.~lO) is linear with respect to the regression coefficients, it is rewritten as,

T where Y is a vector ( y1 Y2 ·· Yr) = ( 81 a1 a2 ··· ap b1 b 2 ··· bq Lp Lh)T, <I> is a regr~sslon matrix and U is a vector ( .!.· f1-Tl·-f2-f2· ·· · , < f 1 -11)2,( f 2-12 )2, < f 1-f1 )( f 2-f2 '· ··· )T.

Applying the least squares method, the esti­ mates of the regression coefficients are given by, where~ CyU and CUU are the covariance matrices and <I> means the estimated value of · <1>.

Four variables are used as the ·articulatory parameters, namely the jaw opening ej, the weighting coefficients of the principal components of the tongue contour a1 and a2, and the lip protrusion Lp, which were introduced in the preceding section. As the acoustic features, first two formant frequencies F1 and F2 are used, because they are the most significant features to be closely related to the vocal tract shape for the vowel-like sounds. The data from which the regression coefficients are determined consist of the samples distributed around the five Japa­ nese vowels and real ones obtained from the X-ray pictures, and the total number of the samples is 300. By utilizing the real articulatory data for the estimation, some cooperative relation between the articulators is included in the regression coefficients. The formant frequencies are cal­ culated according to the algorithm presented in the preceding section.

The estimated result for the synthesized con­ tinuous speech /a i u e o/ is shown in Fig. 4 in comparison with the original articulatory para­ meters. Solid lines show the original articu­ latory motion. The .first and the second formant frequencies at every moment are calculated for this articulatory movement and conversely the articulatory parameters are estimated by Eq.(ll) for those formant frequencies. The estimated values agree with the original ones except for the slight deviation in the tongue parameters a1 and a 2 · However, this result means that if the model is just fitted for the speaker, the estimation would be successful by the nonlinear regression method.

The result for the real speech data /a i u e o/ is shown in Fig.S. The speaker is different from the person whose data were used to construct the model. Although the estimated values cannot be compared with the true ones because the X-ray data were not taken for this utterance, they convey the typical nature of each vowel. For example, vowel [u) is characterized by the most positive value of a 2 and the lip protrusion and clearly distinguished from [o) by the differences of the jaw opening and the tongue parameter a 1 · Front vowel [i) is characterized by the most nega­ tive value of a1 and distinguished from [e) by the difference of the degree of the jaw opening.

In Fig.4 and Fig.S, the marks which indicate the estimated values of the formant frequencies mean the calculated formant values corresponding to the estimated articulatory parameters. It is seen that the correspondence in the formant domain is very well. Nevertheless, there are slight deviation in the articulatory parameters in Fig.4. This indicates that the first two formant fre­ quencies are not sufficient to decide precisely the articulatory parameters in some region.

In a practical point of view, the outputs of filter bank is more convenient than the formant frequencies as a set of acoustic parameters, because the calculation of the formant frequencies is not so easy matter. Then the estimation using the output of the filter bank was tried. The output signals of the filterbank were reduced to 3 or 4 components by the principal components analysis and those main components were used as the acoustic parameters in the nonlinear regression. The result is s hown in Fig.6 for the synth£>s i zL·d vo i ce /a i u e o/. Compared with Fig.4, the accuracy is almost t h~ same . 11H! original and the estimated s pec tral patterns are shown in Fig.7. Formant f requencies from first to fourth are in good agree­ ment.

lt may be concluded that the articulatory parame ters estimated by the nonlinear regression method can be employed as a feature vector for the s peech recognition.

Sentence Recognition Algorithm

Formulation of the Sentence Recognition Problem

In this section the sentence recognition problem will be given a mathematical formulation. Sentence structures mean the categolization of the t ypes of the sentences according to the syntacti c and semantic contents. The method to set the sentence structures depends upon the scale and the complexity of the problem. In this study, from the practical view point, it is assumed that the number of the words is not so large and the language can be described by the context free grammar. The construction of the C.F.G. for the given problem becomes important. It may be diffi­ cult to .find the general procedure. However, if an appropriate restriction for the speaker is settled, the categolization of the sentences is not so hard in a small scale problem.

First, a set of parts of speech W=Wi (i=l,2, ·· ,m) is introduced. The concept of a part of speech may be different from the linguistic one. It is defined so as to include its meaning in addition to its role in a sentence. The j-th word of the i-th part of speech is denoted by wij· so that Wi={wijl j=l,2,··,Ni}. It may happen that a word is regis­ te red in two or more parts of speech. The total vocabulary of the system is .UWi·

Second, sentence structures are described by a context free grammar G 0 (Vn, Vt, P, z 0 ) which gives the arrangement of Wi in a sentence. The grammar G 0 is assumed to .be not ambiguous and

, ..

Vn a set of non-terminal symbols,

Vt a set of terminal symbols,

za initial symbol, P a set of production rules .

The set Vt is the set of parts of speech, i.e. Vt= {Wi li=l,2,··,m}. The initial symbol z 0 means that the state of the machine is at the a-th state. The set of the state Z={z 0 lazl,2,··,c}, where c denotes the total number of the states of the machine. The grammar G 0 produces the sentence structures which can appear under the a-th state. It is assumed that G 0 is not ambiguous. Therefore one sentence corresponds to only one left most derivation and is described by a sequence of the us ed production rules. The set of the sentences which is generated by a sequence of the production rules Pk(a8,l)Pk(a8,2) ··· Pk(a8 , ~ ) is denoted by saa ·

when· Ls shows the leng th of the sentence Sct ll · An element saBy means the y-th sentence with the 8-th sentence structure which may be ari se unde r the a-th state, and can be expressed as,

where i(a8 ,h) indicates the part of speech of the h-th word in the sentence of SaB and further j(aS ,y ,h) addresses a word in Wi. The fact that one sequence of words has meaning is considered as that th~ sequence is one possible SaBy ·

The purpose to introduce the sentence structure is that the strong mutual dependence of words in a sentence is absorbed in the sentence structure and j(aB,·,h) becomes nearly s tatisti­ cally independent in S 1 xB· TIIC ope ration of th<· machine can be shown as Fig.8 that is an l'Xample to give orders for a robot in a dialogue to move forward or turn in an as s igned manner.

Method of Sentence Recognition

Let the speech pattern representing one word be n, which is a random vectors sequence. A sen­ tence can be represented by a string of n as ~ =[ n1,n2, .· , f2L]. When a sentence J is observed at the state z 0 , the sentence recognition is to de­ termine B andy. The optimal determination of B and y in the sense of minimum error is that of

maximizing the a posterior probability f 0 (B,YIJ).

The numbers j (aB,·,h) in B are assumed statisti­ cally independent. Then, Eq.(l6) is rewritten as,

Using the Bayes theorem and taking the logarithm, the objective function to be maximized is got as,

where the common term was discarded and,

The first term of Eq.(l9) comes from the word recognition and the second terms of Eq.(l8) and Eq.(l9) carry information on the context and the situation. Then the sentence recognition problem was given an effective formulation as an optimiza­ tion of Da(a, y I..&) with constraints come from the fact that a and y must be selected from a restricted set.

If the constraint about Y is neglected, the optimization can be performed by using dynamic programming. Let the maximum value of Da(a,j(aa,., h) I.J) at the h-th stage be,

The following recursive formula is derived.

where

From this formula one optimal sequence of words for each a that satisfies La·L can be determined. Therefore, if a sequence is found to fit for any a*.Y* the optimal solution is decided as BE a· and y · Y*· In this method, there is a possibility to appear combinations of j(aa,·,h) which corre­ spond to not allowable or meaningless sentences. In such case the calculated a· and Y* should be rejected and the second optimal solution should be tested.

An important problem arises in the above formulation. That is the aberration of the scene recognition between the speaker and the machine. The speaker does not know the state of the machine or the qpe~ker utters a sentence by mistake ~hat should not be permitted in that situation. These phenomena often occur in the actual conversation. Particularly in the case that the machine made a misrecognition and went to a state unexpected by the speaker, it is difficult for the speaker to do a suitable action. In such aberration condition the above algorithm cannot work satisfactorily. This phenomenon always appear when the relation between the speaker and the receiver is made tight to improve the recognition score.

Conclusion

In this study two important parts of the speech understanding system were considered. The feature extraction method that utilizes the phy­ siological and the phonological constraints was proposed. It will be effective for the speech recognition by improving the word or phoneme re­ cognition score.

Recently, several attempts have been made to estimate the cross-sectional area function of the vocal tract using the state space expression of the acoustic wave in the vocal tract. However, in those framework, it will make the problem too much complicated one to consider the various con­ straints of the articulatory motion. If the dynamics is taken into account in any sense, the dynamic character of the articulatory motion should be considered first and the state space ex­ pression of the acoustic level may be ignored because of the difference in their time constants.

The sentence recognition algorithm is well formulated and very compact. Then, it makes easy the real time operation of the speech understanding without using special hard wares.

The two methods were discussed separately in this paper. But, if the suitable algorithm to decide phonemes from the extracted feature is added, the system will be completed.

The author thanks Dr. H. Fujisawa and Mr. M. Honda for their cooperation.

This research was partly supported by Kawakami Memorial Foundation.