On the Use of a Taxonomy of Time-Frequency Morphologies for Automatic Speech Recognition

1. Introductlon

Most . of the popular techniqucs used today for Automatie Speech Recognition (ASR) are based on compa.rlSlons between prototypes and speech data [1]. Whcn the recognition task involvcs simple vocabula.ries, word prototypes arc uscd. For complicated tasks, syllable prototypes or centisecond prototypcs a.rc used[2]. In the ln.ttcr cn.se, n. comparision bctwccn dn.ta a.nd prototypcs allows one to assign a Iabel to segments of specch having fixcd duration. Thc Iabel of a spcech scgment is one of thc proLotype tbat best mn.tcbes tbat segment. Matehing is usually con text-free.

Wbctber knowlcdge about speccb analysis, syntbesis n.nd perccption should bc taken into account or not in ASH is still tbc objcct of discussions among the rescarcbcrs in tbc field. Automatie recognition of connectedly spoken lctters and of la.rge voca.bularies is still an unsolved problcm.

As an attcmpt to solve tbis problem, a system of plans for cxtracting and using acoustic properties has been proposed and a general framework for its implementation has bccn described[3].

Thc systcm allows to segment continu.ous speech into pscudo-syllabic scgmcnts. Each segment is not necessa.rily a syllable, but an acoustic unit to be described. Some portions of. this unit can act as contextual constraints for thc description of other portions.

Thc purposc of this paper is that of introducing a novel approach for the dcscription of acoustic segments characterized by spcctral lines.

A skeletonization algoritbm is applied to digital spcctrograms. A variable number of lincs witb different durations inside an acoustic segment are thus obtained avoiding tbc crrors and the difficulties of tracking formants. A pattern of spectral lincs is rcprcscntcd by a hiera.rchical description.

I·'or tbc application dcscribed in tbis papcr thcrc a.re only four Ievels in the hierarchy taxonomy but the Ievels

On The Use of A Taxonomy of Time-Frequency Morphologies for Automatie Speech Recognition

Rena.to De Mori and Ma.thew Pala.kal

Concordia. University, Depa.rtment of Computer Science, 1455, de Maisonneuve Blvd, Montreal, Quebec. H3G 1M8

as weil as thc relations at eacb Ievel can be expanded in ordcr to makc tbe ta.xonomy rcliable enough for a givcn rccognition task.

Experimental rcsults on the charactcri~ation of diptbongs in conncctcd digits and letters are discussesd.

2. A Taxonomy for Spectral Llnes

Spectral lines are extractcd with a skeletonization algoritbrn from tbc time-frcqucncy-energy pattcrns obtaincd by considcring tbe 0-4 klh portians of spectra computed witb tbc Fast Fourier Transform (FFT) algorithm applied to tbc prccmphasizcd speech signal. A bierarchica.l description of spectral lines is tben obtained.

Tbc timc-frcquency-cncrgy pattcrn pscudo-syllabic scgmcnt is processcd by algoritbm whose deta.ils are given in [4]. for a givcn a skclctonization

2.1 Tbc dcscription hicrarcby for spectra.l lines.

Tbe dcscription bicra.rcby for spectral lines is based on acoustic properties tha.t a.re known or a.re expected to bc pcrceptually significan t.

The bicrarcby follows a.n open ta.xonomy that ca.n be expanded to incorporate new items and ncw classes.

At levcl-0 of tbe ta.xonomy spectral lines are described by vectors V. of triplets (t .. , f .. , e .. ) J Jl Jl Jl (j = 1, ... J; i = 1, ... , I.) where t .. is a time reference in J Jl centiseconds, f .. is a frequency value in Ha and e .. is an Jl Jl energy valuc in dB.

At lcvel-1 spectral lines are described by morpbology symbols xk E E1 and a sequence of attributes, consisting of time and frequency values. l:1 is an alphabct obtained by concatcna.ting two symbols betonging to alpha.bets E1a a.nd l:1b. E1a describes temporal events and is defincd as

follows:

l:1a. : { A:asccndent, H:borizontal, D:dcscendent}

E1b ~ives a rougb indication of the frequency location of thc mid-point of tbc line:

l:1b : {LO:low, LA:low-avcra.ge, A:avcragc, All:avcragc-high, lll:high, VH:very-bigh}

Notice that level-1 descriptions contain pointers that allows one to exa.ctly pick-up the triplets of values at the levcl-0 dcsription.

Lcvc\-2 descriptions refer to loca.l tem_J.!oral relations of \evc\-1 descriptions representing lines that a.re dose in frequency. They are of the following type:

whcre !R m is a rcla.tion symbol, y m 1 and y m 2 a.re line descriptions likc ak.

lTim symbols bclong to an a.lphabet E2 whose elements a.rc dcfined in Ta.ble I.

Lr.vel-3 dcscript ions ca.pture important frequency rchüion<l in a broa.d frequency range of level-1 and level-2 descriptions. T hey are of the type:

w hcre Z nl and Z n 2 can be level-1 or Ievel-2 descriptions.

Qn symbols bclong to an a.lphabet E3 that contains, at the moment, three symbols.

In this Wil.y, spectral morphologies relcvent for computer perception a.re cxtracted a.nd dcscribed through subsertuent Ievels of abtraction without Ioosing detail of thc orisina.I spcctra.. Fig. 1 shows an example of the description obtained for the dipthong / rei/ in /k/.

a. Hypothesis DescriptionB Generation Uslng Hlerarchleal

Ji;xpressions of predicates whose arguments are elemcn ts of hiera.rchical descriptions are used as prcconditions for actions of various typcs. Some actions considcr, for example, t he concatena.tion of spectra.l lines throu1;h a low cnergy trausi tion with possiblc gaps a.s in the ca~e of / ju/ a.s reprcucnted in l<'ig. 2. Hcrc the dctcction of an FF followed by a BF is a. precondition for scarcbin{; the abovc menLioned Iow encrgy tra.nsition which i~ circlcd in lt'ig. 2. Oncc descriptions at all Ievels have been obtained, then specific parameters relating elements of different descriptions can also be extracted and a.-priori proba.bilitics of them ca.n be collected.

The entire descriptor can be seen as an expert system that conta.ins a set of operators. Operators a.re dustered a.nd clusters are ordered so that there will be a duster of operators for ea.ch Ievel of descriptions. Possible cha.ins of operators are specified by the Expert System Knowledge. The specific cha.in of operators that is applied on a givcn pa.ttern depends on the matehing bctwecn Knowledge and da.ta.

Vector Qnantization (VQ) can be considered as one of the operators making this system more genera.l than the ones just based on VQ.

Part of the Expert's Knowledge is used for hypothesis generation a.nd may contain a.-priori probabilities.

Let <I>(k) be the hierarchical description of the k-th syllabic segment. Hypothesis H(k) are generated by matehing <I>(k) with the system knowledge.

II(k) may contain ambigous hypotheses. For exa.mple, a. vowcl ca.n be idcntified as a front vowel, but a doubt may rcmain whether the vowel is /i/ or /e/. Hypotheses are linked with the descriptions that generated them and a RnmmR.ry about hypotheses and descriptions is kept .

As k increascs, the summary is updated and consistcncies a.re eva.luated and used for pruning ambigous hypotbeses .

For exa.mple it is well known that spectra.l lines a.re relat ed to forma.nt frequencies and that formant frequencics of vowels are among the acoustic propcrties mostly affected by speaker va.riability. In the case of connected pronouncia.tions of lettcrs, it is difficult to distingub h between / i/ and /e/ until a dipthong frei/ is hypothesizcd. At this point, the system control knows better what are the differences between fe/ and /i/ for a par ticular speaker and can put this knowlcdge into the summa.ry and use it for disambiguating hypotheses already considercd. The knowledge written into the summary about a speaker or its mood rema.ins in the summa.ry fra.mc until new evidcnce makes it change.

The idca. of using a 11ummary frame for checking consistencies ha.s been a.pplied to the simple exa.mple that will bc describcd in Section 4. It appears to be promising espccia.lly because rela.xation methods are applicable inside thc summary fra.me thus a.llowing to check consistencies among acoustic data of the same speaker collected in a short period of time.

Ma.intaining in time the belief contained into the summary fra.me is not an easy task bcca.use it is difficult to establish when a. belief has to be considered obsolete. It is ccrta.inly worth kceping into the summary frame acoustic information collected in a frame during the pronounciation of a. scntcnce.

A systcm for inductive learning of discriminent deGcriptions involving acoustic propertics of phonctic events ha.s bccn devdopcd[5]. This system is based on principles pn:sented in [6] and ca.n be adaptcd to the case of hcirarchica.l descriptions.

4. Experimental Results and Coneluslons

The experiment reported in this scction refers to the use of hierarchical descriptions for improving the recognition accnracy of connectedly spoken Jettcrs betonging to the so called El set defined below:

From previous experiments reported in [3], the confusion bctwecn /k/ and thc other elements of the set is responsible for more than 10% of the overall error rate. All /k/ is the only Ietter containing a dipthong, it is expected that the detection of the dipthong /~ei/ will improve the recognition of the El set.

J<'or t.his purpose samples from 5 anglophone speakers were selected (3 male and 2 female}; each one pronounced 20 scquenccs of fivc Jettcrs each.

Knowledge for the /rei/ of /k/ contains an FF with a descendent line or a followB-down feature on the first elemen.t of the rclation and an ascendent line or a follow-up fcature on the second element. For the other lcttcrs, the /i/ vowcl was characterized by the FF feature without the followB-down or descendent feature considered for / roif.

The results shown in Table li allows to reduce to zero the errors involving the Ietter /k/.

The strong evidence of follow-down feature and 100% presence of FF in Ietter /k/ allows to distinguish the dipthong /rei/ and to unambigously recognize the Ietter /k/. Table II shows also the frequency intervals in which spectral lines of /re/ and /i/ involved in the FF relation were dctected. AB these intervals overlap, it appears doubtful that context-free recognition algorithm can be cfficicnt in a multi-speaker detection of the dipthong /rei/ as opposcd to thc vowel /i/.

Pronounciations of connected dipthongs in { l,I,U,9,5} pronounccd conncctcd wcre analyzed. Sentences from 5 male and 5 fcmale speakers were considered.

Temporal relations involving disjunctions of conjunctions of descriptions were infered. These relations allowed to correctly segment and unambiously characterize more than 90% of the da.ta. Work is in progress for disambiguating the most difficult data. a.nd for a.na.lyzing more speakers.

Thc results obtained show tha.t heira.rchica.l descriptions a.re powerful tools for detecting recognizing dipthongs as opposed to single vowels. resea.rch will continue towa.rds the goa.l of multi-speaker recognition of connectedly spoken Jettcrs numbcrs. a.nd The the a.nd

Ack nowledgements

This work was suppor tcd by the Na.tura.l Science Engineering Council of Cana.da.. and

Di bliography

1. L. R. Rabiner a.nd S. E. Levinson lsol a.ted a.n d Connected Word Recognition. TheorJ a.nd Sclected Applica.tions, IEEE Tra.nsactions on Communications,voi.COM-20 . no. O,pp. 621-ßOg, 1081.
2. L. R. Bahl, F. J elinek and R. L. Mercer A ma.ximum likelihood Approach to Contl nuous Speech Recognition, IEEE T r ans . on Pattern Analysis a.nd Ma.chine Intelli gence, vol. PAMI-5, no. 2, PP· 11o-1go, 1083.
3. R. Oe Mori , P . La.face and Y. Mong P arall el Algorithms for Syllabic Recognition In Continuous Specc h,IEEE Trans. on Pattern Analysis a.nd Ma.chine Intelli ge nce, vol. PAMI-7, no. 1, J anua.ry, 1U85.
4. R . Oe Mori and M. Pal akal On The Use of Computer Visi o n Techniques for Automatie Speech Recognition, proc. IEEE-CVPR'SO, LA, 1080.
5. [o] H. Oe Mori and M. Gilloux Inducti ve Learni ng of Pho netic Rules for Automatie Speech Recognition,Proc.CSCSI-84, London,Ont., pp. 103-105, 1U84.
6. H. S. Micha.lski, J. G . Ca.rbonell, and T. M. Mitche ll eds., Ma.c hine Learning: An Artificia.l lntelligence Approach, Tioga. Press, Pa.lo Alto, CA, 1ß83.

Fig 2. Enmple of Conca.tena.tion of Spectral lines.