--- title: "A Speech Understanding System With Learning Capability" authors: - "Renato de Mori" - "Silvano Rivoira" - "Angelo Serra" doi: "" url: "https://ijcai.org/proceedings/1975/74" pdf: "https://ijcai.org/proceedings/1975/0074.pdf" venue: "Proceedings of the Fourth International Joint Conference on Artificial Intelligence" year: 1975 pages: "468-474" track: "Speech Understanding" keywords: --- # A Speech Understanding System With Learning Capability Renato de Mori, Silvano Rivoira, Angelo Serra *Proceedings of the Fourth International Joint Conference on Artificial Intelligence*, pages 468-474. ## Abstract A speech understanding aystem w ith learning capabilities Is presented, lis relevant aspects are : a) the spoken sentence Is repre&ented conc i sely by a description lhal can be used 10 reconstruct the sentence and to verify whether its meaning was not degraded by the coding. b} Syll•blea or bro•der coarliculatlon segnents ef"e the smalleat units . c) The evaluation of an hypothes is Ia baaed 011 the probabil ity that a syllable. a word or the sentence can gener ate the spectrogram of the spoken message and th at a syntact ic structure can gener11te Its pitch contour . d) Coartic:ulation effects are described in terms of patlern grammars, ~~nera t :n g all the possib le formant trajectories for a g ive n ullerance . e) Spectral and prosodic featu res c an be learned by In ference of stoch~stlc f inlte- state-automata. f) More for,ant choices are allowed for a single syllable segment .. and an algof'lttwn Ia provided for ass i!J"ing to each cholco e probability. ## Full text ## WITH LEARNING CAPABILITY R. Oe Mor l - S . Rivoire - A.. Serra latituto d l EleUrotecnice - Centro per I'Eieborazione Numerale del Segnall - lslltuto Eletlrolecnlco Nazlonale Gall leo Ferraris - Politecnico - Corso Ouce dagll A.bruzzl. 24 - 1012V TORI""' (ltoly) A speech understanding aystem w ith learning capabilities Is presented, ## lis relevant aspects are: - a) the spoken sentence Is repre&ented conc isely by a description lhal can be used 10 reconstruct the sentence and to verify whether its meaning was not degraded by the coding. 2. b} Syll·blea or bro·der coarliculatlon segnents ef"e the smalleat units. - c) The evaluation of an hypothes is Ia baaed 011 the probabil ity that a syllable. a word or the sentence can generate the spectrogram of the spoken message and that a syntact ic structure can gener11te Its pitch contour . - d) Coartic:ulation effects are described in terms of patlern grammars, ~~nera t :ng all the possib le formant trajectories for a g iven ullerance. - e) Spectral and prosodic features c an be learned by Inference of stoch~stlc f inlte- state-automata. - f) More for,ant choices are allowed for a single syllable segment .. and an algof'lttwn Ia provided for ass i!J"ing to each cholco e probability. ## lntrodut.:tion Large efforts have been devoted '"the last years to the des ign of speech understanding systems (SUS) [ 1]-: [l }. revealed many aspects of the problem that are still open and whoae aolutlon Ia · necessary condition for the success of such dn lgne; . The purpose of thla paper is to present e fremework In which eome of the above probl.,e can be eolved with successive reflnementa. Particular care is devoted to the probl.-n of emitting and verifying hypotheale at the syllab ic level; this takea Into account. aa much ·· possible, the knowledge eoout the structure of speech that ia available to a phonetic ian performing a "isual inspection of spectrograms. On the line of 11 previous work [ l ], coarUculatlon effects are tak·n into account by a grammar of ·peech th·t is Inferred by an automatic procedure. Concepts from phonet ics. syntactic pattern recognit ion theory, grammatical inference, stochastic automata, informat ion theory. Ident ification and modell ing. 11nd natural language proceaslng theories are uaed In this project to <.haracterlze sources of knowledge and data bases at d ifferent level!i of the processing, namely parametric, acoustic, phonological, le.w.lcal, S)ont4Ctlc and semantic. The most relevant and of" iginal aspects of this project are the fnlluwin9: - .1) The "Pokcn c.entence Is represented concisely by a description that can be used to recon:~truc.t the sentence and to verify whether Its meaning was riot degr.5ded by the coding. - b) Syllables or broader cC\articulatlon segments are the sm11llest units. - c} The evaluat ion of a hypothes is is based on the probability tMI a syllable, a word,or the sentence Clln generale the speclrogram of lhe spoken message and that a syntactic structure can gener ate Its pitch contour. - lJ) Coart iculation effects a r e descr ib~d in terms of pallern grammllr"S genef"at­ ing all rhe possible formant trajectories for a given utterance. - e) Sp<"c.tral and prosodic features can be learned by Inference of atochast lc f inite-s ta te-eutomata. - f) More formant choices are allowed for a single syllable aegnent end an atgorlttvn is provided for ass igning to each choice a probabi lity, 1 he ory·ni.a:ation or tile system and the hypothesis evaluation al gur ittvn, operatino at the- syllal.J·c le\lel, allnw one to recover lnfot·matlon tu·t d·e to ert'OI'S in s.og~nentat ion, spectral estimates, for mant tracking and fornliUlt coding, lhe ayslem hos been desinned also to be used to investi!)ate some I' I" ·Lllt .... ~ ·s r"ela led to fields other than SUS, like speaker charu(.ttrization , lnveallgallon of phoneUc rulea In contlnuoua s~ech and modelling of human speech perception. It has to be conaldered more an Instrument for research than a commerc ial Implementation of a SUS. ## The acoust ic level At this level the Information about the sPoken sentence Is the time waveform. II Ia sampled at 20kHz and processed for e111.tracting Jhe amplitudo and pltch,uslng an algorithm based on comb-fiherlng and patlern recognition Furthermore some global features are e111.tracted by a bank of digi t.JI filter· opef"at ing In real-time. The time evolutions of such par.melers are descr ibed by a very simple language that is proceaa~d by a set of f inite-stale automata (FSA) that locate ailences (SL). unvoiced tracts (UT) . vowels (V) and tracts of successive voiced consonants (VC). The details of the detect ion of such "elemental fragnents 11 are reported in [l] and the descr iption of the special purpose that performs d igital filtering in real-t ime c11n be found In { ·J. The sequence of elemental fregnent· Is then processed by a simple FSA that locates 1he speech segonents · . c.elled pseudo-syllabl~segne-nts (PSS.), wher"e coarticulation is e.w.pected toLe more preOominant . Not ice thai the PSSs can overlap, The preliminary seSJnentalion has two tasks. - I - It dellmlta the voiced tracts of the speech waveform on wh ich the spectre are computed, using pltch- synchronoYa linear pr"edlction, while for the unvoiced tracts an asynchronoua FFT Ia performed. - 2- The preliminary aesrnentetion dellmils the fields of spec1rograms corr·spo!l ding to PSSs , in which possible formant paths are specified and organized in a multi linked deta atructure. Furthermore thla aegonentetlon anc:hora on the ateady-slate portions of vowela, lhe starting po inte of the eatlmatea of the best possible formanls. These formants are uaed to emit the first hypolhesta about the possible ayt lables present In the spoken sen1ence, whi le the requeat for · h)·pothea is verification from higher levels f"eaults In a sear"ching for paths In some specific allowed passages. This aearch ia carried out In the' multi linked da te structure and if possible patha af"e found, the If" probabll ltlaa are also ct'lmputed. A node in the data s1ruc1ure contains information abOYt a ff"equency interval of a gi\l<'n spectrum, where a concentration of energy ·nd lhe maximum d this energy ha ... e tern found. The organization of the data structure and tho elgor lthma for f\Jrmant track ing are reported in [SJ. The way in wh ich probabilities are assigned will be de5cribed briefly at the end of thla paragraph. The posaibility of performing pitch-synchronous FFT on the voiced tracta is also allowed. The difference between FFT and llne·r prediction (l.P) In thla application is tha t FFT generally Introduces many apurloua paths In the spectrogram. while LP r"equirea a careful operation of gap-fill ing; both the aspects are trea ted by the formant tracking ai!)Orlttwn, bul one method can be better or poorer than the other. depending on the waveform. Formants, amplitude and p l!ch ·re presented a t lhe higher levela as a lis.t of desc:riptionti; the form of these descript ions are lntroc:kJced In lhe follow·n~ account . ## Ampl itude descriplion (AOES) The amplitude deacr lpt lon haa th .. primitive ferma rep··eaented by the follow ing nun-terminal alphabel : These forms. are related to · terminal alphabet containing tt.rH s)o·nbots representing a linear apro.w. imatlon of the ampl itude curve; lhe lermln.l alphabet iii-: and the compoaltlon rulea ere: The cc.nposlllona are performed b~ a alrnple flnlte-atate automata'\ operallng In a lime close lo the real-lime. Each primitive symbol Is followed by four altrlbutes: the clJratlon, the amplllude of the first poinl, the abscissa and the amplitude of the maximum. These per..-netera allow one to reconstruct the ampl itude waveforms with polynomial approkimat ion. ## Phch oJescriplion (POE.S) Pitch descripllon is very similar to the amplitude description , except that a new symbol is provided for the silences or lhe unvoiced lracta. ## O"scrlptlon of apoctral features for an enllrelv voiced PSS The description SDES of an entirely voiced PSS Is characterized l.ly the following rules: where FIT Ia a tract where I formant linea have been detected. The descriptions of such tracts are: FIT · (FtC) (AID) (b) F2T · (FtF20) (AIA20) F 3T · (F tF20) lA IA20) (FlO) (AlO) F.T · (FtF20) (AIA20) (F3F40) (AlA"!>) F5T · (F tF20) (AtA20) (FlO) (AlO) (F4F50) (A4A50l Where FtD Ia the descript ion of ijle time evolutions of the h-th forma'11 frequency; ~. ·0 is tt':.,. ~SC:"'!ptl:o:-!"1 of tho tl,...e evolu!lc·,· nf the h-th formant amplitude; (b) Ia a set of symbols describing the 1 + 10kHz speclrum of voiced fricative and voiced atop ~unda; F iFjO Is the description of lhe evolullon of F 1 and FJ In the F 1 - FJ plane; A IA jO is the descrlplion of 1he evolutions of the emplltudes A 1 and AJ In the A i -AJ plane. The descr iptions have two types of primitive forms, namely lines and etabl· zones . These primitives are deteCted using an algor i thm presented In {e.] A stable zone corresponds to a quaal-slatlonary portion of the speech waveform and is represented by a symbol S with the following allributea: the duration and the gralo'it) center coordinates of the points belonging to the stable zone. The lines are represented In the SUS by one of eight slope s~mbols, k, 1, m, n, o, p, q, r, corresponding to angular sectors of 45·, followed by two numerical attributes: the duralion and the line length. The cho ice of the maximum allowable error In lln·ar approximation i 5 a cc.mpromise between compression and fldellly. This value certainly need not fall below the limen· of the perception of formanls, that Is ll of th111 formant values. The maximum error has baen selected In order to oblaln lhe detection of a single stable zone for all the non reOJcad vowels in a corpus of 50 5ent~nces . Values of 70 Hz for the first formant, ISO for the second and lCXl for the third, have been f~nd. Assuming a gau"slan distrlbullon of the errors o"er the stable zone, a total average value of the errora over the three formant regions of sZ i s oblained. Simi lar results are obtained for the linea. ## Silence and un"oiced tracts description Silen"cus are described by a symbol SL and Ita duration. To represent the unvoiced tracts, the symbol · liT is used 1 followed b~ a concise descr iptiun of an average spectrum of lha unvoiced !le:gnent. For s~pftr,t lng two successive frlcatllo'ea an analysis of the spectral def'ivate Ia performed, according to pre"ious elq)eriencea In lhe aegnentation of continuous speech. Nasals and nasalized vowels often geMrate a aegnent o f depends on frequencies and ampliludea of the firs t formant . se9'"ents for wh ich this poss ibil i ty i s detected, a more accurate re$eorch ror clusters or other formanl l ines is started, Two tracts having a different number o f formants can be con<alt"lla ted In different ways ; these situati ons 4re de·cribed Introducing pred ical"!'o t-etween formant descriptions r l}. The descriptions obtained in this way allow one to reconstruct and l is ten lo the spoken sentence and to evaluate If the sentence st i ll pre:ser"es it& meaning. ## Probabil ity ass19f'!!lenl The influence of segnentation affects only the prel iminary emission of hypotheses about poss ible phonetic transcript ions of PSSs. In this operation and In the complementary operation of hypothe1 i s \ferlflcatlon It i s important to know the probability P (d 11 /a 1 ) that the descr iption d 11 truly represents the ae!Jnent U 1 of the spectrogram of the spoken aeneence. The subscripts of such parameters are to be Interpreted ## follows: d 11 : Is the 1-th poss ible description of the 1-th spectral sesrnen1; I · I, 2, ··· , 1 1 , whera L Ia lhe number of segnents evidenced on the spectrogram at a ce:rtain lnstanl of . the undet"atandlng, 1 1 Ia lhe number of possible descript ions of the I- th aesrnent . The probability P (d;,Ja 1 ) will be used In the next paragraph for the e"aluation of e syllable hypothesis. 0 1 ; 1hua : Let F 1 be the concatenation of .11 1 paths ft 11 , n 12 , · ·· , n 1 tJ 1 , where each path ends at a node where It can be followed by more than one way. Then one has : The generic conditional probability In (2) Ia the probability lhat g iven a path n 11 n 12 ··· n 1 ~ on 0 1 for the flr5t formant, this path cont inuea w i th 11. 1 JJ+I' This probability Ia evaluated cons idering the average amplitude , lhe number and duration of g·ps and how far It 15 possible to go follow ing V-t 1 , instead of the other poss ible direct ions. W i th relations similar to (2) the other conditional probabilit ies In ( 1) can be cornpuled, remember ing that, given the first f formants, ttl· (f + 1)-th should be In a proper frequency range and above the first f forr114nts , This approach allows one to use the information about formant avolut ions that hav e been proven very significant by meny perception experiments. Sever"! possible formant patterns and the i r probabilities can be obta ined avold ing that an error in formant tracking comprom i se i rreparably the ·uccess In recogn1t ion. ## The eylleblc le"al At the syllable level lhe information about the tp~tonee is the descr iption d 11 of the lime evolutions of formants and amplltudes 1 together with lhe probabilities P(di/g 1 ) of lhe description dil · The a- priori information source for lhe syllable level Ia a grammar of ape~tch generating a lunguage reco!:lnized by a eel of slochastlc - f inite-sta te automata (SFSA), w i th some auxiliary units that will be descr ibed later on. The formant pauerns corresponding to a g iven VCV utterance pronouneed in various con1exts cannol be random patterns ; they have to respect some rules under I~ 1ng the relations betwec.n the art icu l atory commands and the spectra of the generated wav eforms. On the other hand, the patternto ar e not equal bec au'le 5ome d is tor t iona on the archetypes do not alter the interpretat l - on given by a listener to the or iginal utterance and it i s important to k~ow the atatlatica of auch dlatorllona. Tt·aua the SFSA are a proper tool for reprea­ entlng the poaelble pauerna of a given utterance.· Their Implementations end ahelr learnln· are a leo feeaible because the ses;rnents considered are short in t ime and their "oeslble petlerna are deacrlbed bye concatenation of few aymbola belongln!) to a very .mall vocabulary. Two opera lion· are performed at the ayllablc level, nemely hypotheais emiss ion end verification. At the beginning of the under a tanding, the sesrnenaer represents the spoken sentence as a cont inuous sequence of PSSs. This sequence ia the f i r~ t sesrnent·tlon hypothesis. Then for the voiced portions of each PSS, the most likely formants are extracted and a descr ipt ion ia generated. The description· are processed under the contro l of a grammar of &peech and are lranslated Into poa&ible phonet ic transcriptions, with associated the probability P(S/111 1 ) that the syllable Sj corresponds to 0 1 , the 1-th segnent of the spectrogram of lhe spoken sentence. The grammar of speech Is a stochastic grammar r~present i ngthe poaal~le panerns for each coarllculation instance, corresponding to the concat­ enation· of elemental fragnenta generated by the segnentation grammer. The starting aymbol of the grammar of speech is PSS; lhe terminal alphabet contains all the aymbols with which the deacr iptions are made; the non­ terminal alphabet containa all the possible concatenation· of phonemes for wh ich coartlcutatlon affects the formant patterns ~ven after the description approKim­ allona. The non-terminal alphabet contains alao the a)'mbols emil led by the au­ .llllllary unite preceding each SFSA, and acttno ·· lranalatora of lhe.,cteacripl­ iona made of symbols, attr lbules and probab i lities Into i.ymbols and probabilil­ los, prov ided thai some relations hold between the attributes. F iMIIy , the produLIIons of the grammar of speech are r ight-linear stochast ic rewr iting rules, der ivable from the SFSA inferred by the procedure that will be presen­ ted In the neKt paragraph. The operation· of hypothes i s em laalon and verlflcalion contain hy~ ottw. .. IA evaluat ion as a comnonent : thua this ooeralion w i ll be descr ibed firat. Fig. 1 - Sc~e for hypotMsla evaluation. ![Image](https://ijcai.org/proceedings/1975/fulltext/74/image_000000_94cfd05aefda53415609d0b4c756dc6eb1d9b24340346ea07cf9250ea380ebc7.png) ## H)'pothetlt evaluation Hypothesi· evaluation Ia performed wllh 1 procedure whoae block d iagram Ia shown In fig. I · When a ayllable or a coartlculatlon 81fiJ"Int Ia hypothealz.ed on · certain portion of the apecarogram, the SFSA of that tefiJnenl and lhe correspo­ nding description translator are buill up from !he gr.wnmar of speech. This operation Is s imple and fasl due to the strai ghtforward reletions between the productions and the automata. The tylloble description I· processed b)· the deacr ipt ion tranalator (OT). The OT tranalates a pr imitive foJrm, with ita attr ibute· to descr ibe correctly aome formant evolut ion!, into a S)'n\bol and lla assoc iated probabilicy. The symbol Is emitted only If some conditi ona on the aur i butes ere ver i f ied. These cond ition! are stored for a talker and a typical lexical position of the segnent and can be modified by aome spe,1 )..rr dependent modifiers, act ing ma inly on the formant loc i for the stable-zones constraints. In addi tion, some leKica l-dependent modifiera, acaing ma inly on the durat ;ons , change the const r e in·· In accordance wilh the streas of lhe se~H!nt, 1'he des ign of such modifiers that al~o involv ea the l<nowled!)e of ~.-obability distr ibutions Is actually I imlted to spectral loci and the infiiJence of J I · the streas on the ct.lratlona, pitch and emplltudea. Ia. tot of ln~stl .. tlons are still to be done for characterizing the Individual Influence on formant evolut ion and loc i for syllablea emb"dded in conlinuous apeech and the approach followf'd here should be conaldered a first approKirnation. The input to the o ·r Is the descrlplion dil . lhe output of the OT Ia a string g 11 of symbols w i thout et­ tributea obtained from d 11 wh ich I s a atr lng of aymbola with attr ibutes ; toge­ thet" w i th g 11 , the probabili ty P(d 1 / g 11 ) i s obta ined. F inally, the gil ·s processed by the SFSA and, If the gil i a recogn i zed, the probabilit)' P(g 1 /Sj) ia g i ven. All the obta ined prot».bilil ies are processed by the follo­ w ing algor i tlvn ASP, tha t o i ves P(S/~ 1 ) . ## Alsorithm ASP Remembering that the ayllable tiona recognized by the SFSA, I, · : S J It th.e union of all the deac;rl~ where GJ · { gJ 1 , ~ , ··· , < , ... , v!J Ia the eet of all the deacrlptiona recognized by the SFSA of the ayllable Sj ; the following t"elatlon can M written : Combining (l) and the follow ing relat ion: ## one oe·· : where P(~ / 5jl eon be obtolnod ot the output of lho SFSA when ~ oro recognized. be the aet of the poaslble .S..crlptlona of the apectral aegrnent o 1 and the subset of deacrlptlons of "'I lraMiated by the OT of the ayllable SJ Into g! · Lea Mk . be lheir number. J j Lei Dk be the sel of all the poaalble deacrlpllonathat can be tra~ slated into g! ; let Nkj be their number. Ae the deacr lptlona ere di ajolnt ele- hut : The description! obtained by o 1 muet belon11 to the aet 0 1 and thoee that are mapped into s( and are obtained by g 1 muat belong 10 c! 1 ; thua (7) reduces to: whe:e the P(d! tm/g!) are learned with the SFSA of the ayllabta SJ and the P(d~ 1 m/W 1 ) are obta ined from the c»acrlptor. Using the relationa obtained ·· far, the evaluation of the hypothea la th·t the syllable S j is the trascrlpllon of lhe portion g 1 of the a.,ectrogram, can tte computed ·· follows : In practice, II Is ei(J)ected that often only one deecrlptlen d il Ia recogni~ed and translated by the DT into a g 11 thai Ia reco901zed by the SFSA; In this case the hypothesi· Is simply ev· lualed ·· followt : where the three conditional probablllllea are given respectlvel)l at the outputs Clf the deacrlpaor, the DT and the SFSA.. Let the description dil be recognized as PSSs 5 1 , S 2 , ·· · Sj, ·· · · · ,Sw · The probablllt)l P(d 11 )can be computed as follows : because s,. 52' ··· , sj · .. . , sw are disjoint lleta. Each eddand In f12) Ia obtained from the a-priori knowledge of lhe S)llllen'11 ·· follow· : The probabil i ties In (13) are the a-priori lnformatlona lhat need to be learned, There are several poaaiblaapproMimaaiona for p(di/ g 1 j) that corr­ apond to different reG..Icllons In lhe compulat lon off II) end ( 12), The s implest approMimat ion conslsta in considering F(di/9ij) indep­ endent from j, A bel1er approMimatlon, that aaama to be ver)l realistic, conaista In cons idering the probablllt)l of an allrlbute of d 11 uniform over the range al­ lowed for thla attribute, in order 10 rranslata d 11 lnlo glj · In lhis case, ono need learn onl)l the Intervals allowed for the attributes or dil. F inall)l, 1he eM.sc t dlstrlbuelons of lhe altrlbutes of d 11 ovor lhe allowed intervals of the OT could be learned and used lo compute P(di/9ij). Thi s last approah would re­ qu i rt= a ver)l large number of el(J)erimants, probebl)l w i thout greet I)! improving the ayatem performances. ## Emlatlon of hxpolhesea When a PSS haa b"n described, lla daacrlptlon Ia proceaaad in ordet· to decide whether or not It balonp to the lens)uage generated by the grammar of apeach. The grammar of speech Ia ambiguous because a descriptiOn c.·n be venerated by man)! a)lllablea; thus It me)l be recognized by dlfferer.t SFSAs.This Ia one of the reaaona why the llngulsllc approach prevlous1)1 foll­ owed { 3 J has bean eMtendad now to the field of slochasllc grammars. In pra­ ct i ce, it Is l ime- consumlnv to tr)l to find If 1he descript ion of a PSS Is recosr n i .zect' by any among all the SFSAs of the grammar of speech. For this reason, th" PSS cJesc r ipl lon Ia preprocessed by the FSAa shown In flv. 2 , The outputa o f these aulomata define the aub-aat of SFSAa that ma)l accept the unknown · de-.crtpt ion. Onl)l thla subset Ia used to process the Input deacr lpllon ; the S't" lleble correapondlnv to those SFSAa . that accept the deacrlpllon w i th an as­ soc i a ted probab i lil)l Ia emitted ash)lpotheala of the a)lllab lc level . The set of p r obab i lit ies emlued by the SFSAa, allows one to compule the total probabillt)l that the descript ion belongs to lhe lang~.~age generaled b)l the speech. grammar of From the PSS description, a flral formant descr lpllon tFFD), · secor.d formant descriplion (SFO) and the emplltude description (A IA2DES) ere e ... trectod and processed separately. The automata of fig. 2 are deterministic. 1 he f irst automa·on (FBA.: fronl-back automaton) proceaaes FFD r tecogni zas If there ia one or two vowel a in the PSS and If the)! are front {F) or back (B) · . The second automaton (t-tMLA: high. middle, low automaton) recosr n i zes i f rhe vowels are high (H), middle (M) or low (L). The th ird aulomaton recogni zes It the consonant between the vowe­ ls ; · naaat (N), YCJ iced slop or frlcatl\le (VSF), liqui d or glide (LG), or other (C). Th e automt~ton may recognize more than ono of auch clasae· because of the aml.t iguity of the consonant aegnent or of the presence of lwo consonants. Such automata are learned w i thout probabilitlea,u·ing the aame proce~re presented In the successive paragraph. ## Ver i f i c-at ion of hypotheses When an hypothe5 i s about a a)lllable Sj has to be ver i f ied on certain port ion of the descr iption, the cMscription is processed by SFSA of S J ; If a port ion c1f lhe description i s recognized, the r ecognition pro~bilit)l Otherwise, possible insertion&, omiss ions or aubit i tullons on the descr ipt ion are conald-tr ed In order to racoon l ze lhe s)lilable. Ia emitted ·· a ·n·asure of the verif1catlon. ![Image](https://ijcai.org/proceedings/1975/fulltext/74/image_000001_e5cb3e6dab6a73ed2791bbfd09169e6f1cd4aa4ce3fa051302390034fd6c7aee.png) The descr ipt ion modificat ion.- are then verif ied in the spectral de­ ·· base and if the)! are found poss iblelhe l r probabil i t ies are compuled and the ·set of modi f icat ions which make tho h i ghest probebilily of recogni t ion of the a)lllable Is accepled, The correspondi.ng probab i lity i a"assumed to be the measure of the ver i f icat ion. If these modifications aller the prev ious ae!)'T'Ientat ion, new hypotheses are gener41ed at the a)lllabic level . If some of the descr iption modifications are found to be Inconsistent wllh the spectral date base, e loss funct ion Ia cornpuled that lncraaaea with the un i ls of Informat ion of the events that are not rec·lved at the pr oper time l.ly the SFSA, If man)! so lui Ions are poas ible. 1he one corresponding to the mi­ n imal loas Ia chose-n. ~·earning Ia accomplished through the conalructlon of a atochaallc f l nit~state aulomaton for a fini te set of stochast i c strings of symbols descr i­ b ing the same syllable {8], Ever )I string with probabil lt)l o f being d i fferent from zero Ia gene­ rated b)l the grammar estoclalad with the automaton, while none of the nlher str ings can be generaled, The samples of Information are presented aequen11all)l to the Infe­ rence ntach ine, which determines a .aochaatlc finite-alate automa lon after rha 1-th ob· ervation on the basis of the t-lh aanple end of the automelon generated af ter lhe .. (t-1)-th observation. The probabll i t)l of generation associated with each atring Ia obtai­ ned by 1he relative frequenc)l of occurrence and i s approached aa t eppr"oaC".haa infini te. The tearninv a)latam Ia baaed on an Interact i ve procedure with a human trainer . The teochar checks for the correspondence of the air ing to an acceptable phonetic dascr ipllon of a g i ven a)lllable and decides If It muat or must not Ue learned b)l the inference machine. He can be helped to dec i de b)l l istening to the ·ynthesla of the ·II ... tabh~ obta ined wllh tho phonet i c descr ipt ion paremetora. The s1ochast ic finite-alate aueomaton satisf ies two condit ions: It Ia unamb i !)uous and it has lhe nlinimal number of states. A ·tra lghtforward proce­ dure generates from it a right-linear grammar with a minimal number of non term inal a)·mbo ts , The grammar Inferred can be conr lnuoual)l updated dur in!) recognl ­ l ion by adding new state· and trans itions 10 the SFSA if a configura tion Ia not rec ogni 2ed. or by ref in ing lhe probab i l it y ··ai gnments i f the conf igurat ion Ia recogni zed. The SUS can in th ia way updale cont inuou s I~ II· knowledge, A me­ a&uremenl of the slab i tit y of th i s knowledge i s obla ined cons i der ing a sy llable as en informat ion source; the possible spectral configurat ions of the ay llal.lle ar e the messages em i ued by lhe source, As the probab i lit)l of oach message can be ob ta ined by the SFSA, the entrop)l of the source can be measured in c.Jifferent momenta of the learning. When the entropy remalna quaal-atatlonary aflora large enough number of el'q)erimenta and no new configurations are learned, a good degree of know ledge has been reached. The learning of syllable confi g...~rations Is performed In accordance wilh the theory of mechanical inference of SFSAs / 7/. The following aspects clarify why the learning of syllable. features is practically feasible and do not requi re a large amount o f memo ry and computation. - 1) l he strings to be learned ha ve a lim·ted length. Neither recuraion nor ccnt·atenellons of the same symbo l can be present. - 2) 1 he term inal alphabet has few S)'rnbols. - 3) The syllable pallerns should respect the phonetic structure of the syllable. Thus the "·a riety of descr ipt ions a c cepted by a syllable automaton canrof.lt be ve ry targe,evun If the number of panerns can be very large. In fact each l ine c an be generated by concatenation of elemental arcs connecting two po ints and the number o f suc h arcs can be largely variable. A deta i led de&cr lpt ion of the Inference procedure is descr ibed in another paper (a} and Is only summar ized in the following. Let r~ ~n) be the S FSA of the syllable 5 j after n aemplea have been learned. Let rJ(n) be the co rreapondm!J non-stochastic automaton.,Beca­ use of the aUsenco of recurs ions in rJ(n) it is easy to construct a corre~pon­ d ing ranular e xpre5t ion R 1 (n). Let l(n·1) be the informati on to be learned bt'!cause it is not recogn ized l>y rj(n). A new regular rx~.Jresslon : ## Rj(n+ 1) · R 1 (nl + l(n·1l Ia considered and the new automaton r J(rl'f' I) is construcled 1 c0naiderlng the der ivatives of RJ(n·t) w i th respect to all the first a-,·mbols, the derivatlvt!a of the derivatives with respect to the ir first symbols and s o on.Thentheprobabil ity distribut ion on 1he automaton is computed. Th is operation Is performed eas ily, because for each transition in the stored automaton the total number of time~ thia transition has been executed i :~o stored with the transition itself. The occurrenc ies of the transitions in the new automaton are computed from the kno\('dge of the old ones. The state minimizat ion problem is s implified by the fact thet no recur,. ion& are allowed. Thus all lhe states from which the same final state is r~ached , w i th the aame atring:l of length 1 and wllh the same probability, are equ ivalent and can be merged. This considerat ion can be eKtended to mer~ the s tates fr om whi c h the same state Ia reached w i th same strings w i th the same Wdh the above cons iderations, the research of equivalent states t an be perfo rmed ae~uenttally, going from the f inal state back to the Initial one cons1dermg tails of strings of increasing length. Fig. q shows the automaton o f the p!o> cudo- ayllabl~ "uiiiA "; the operation of the corrisponding trans Ias ~ ·mr.ur · zed in tab le A 1. When the au tom~ton co rrespond ing to a g iven PSS has been lear­ ned an a l!],.,,.., ,tvn descr ibed in [ 6} i a applied to check if the considered PSS c an he: subd i\. oded into acoust icall y Independent units . Furthermore 1 many auto­ rn,,ta, corr t· ~pund i n ~J to s yllabi'""'!> whose phonemes d iffer fo r few d istinct ive fealures , ha\'e co nsi deraOie ~rt ions In common (e. g . the formant amplitude patternl . These automata can be reduced to a s ingle one where some translalions or w ledge at the phonetic level is characterized by stochastic rules that will be ref,ned dur :ng learning when more insight in the propert ies of the auditory patF inal ly, the stochast ic rules part ially refer to different pc s s· b ili t ic '!> o f articulat ion and part1ally to the noise introduced by the algor­ i thrns that transforn1s the speech waveform into auditory patterns. The noise t-ffects .. r e probably independent from the parlicular syllable compos itions and cou ld be characterized by a s ingle grammar induced by error trasformations us in~ an al go.)r i thm proposed by Fung and Fu {9 J . ## The le101ical. semantic .,d syntactic levels These levels hove been conceived In a almple way and will be de5crlbed briefly. The protocol used so far was Introduced only for experlmtt!! tal purpotoe and refers to on automated reser-vation and Information ·ystem for tra .,~ls. The vocabulary Is actually of about 200 words and the syntaM and SP.<manti c s are described by liim~.Jie netwo rks. The semantic network contains .1 smo.~ ll set of key worda for eac h node and Issues the ma in requests for hypot ht~~ i o.; verificat ion. The lexicon contains a stoch··tlc flnlle alate automaton for the ruot of each word, corresponding to the different ways the word con be Pr0f"2 unc:ed. The probability asalgnment Ia strongly dependent on the speaker and his dlalacti the poa&lbllltlea for each word are few because the llallan L 'lngua~ e does not allow a l.,.ge variety of realizations . The different pronunciations taken Into acco.unl refer to utteranc~s that are perc~ived as clearly different dur ing the learning of auch rules. Prob lems o f alterat ions In word concatenation are el.a very limi ted In the Italian Language, The syntactic network refines the hypotheses in order to c r «! olte or recognize a sequence of words that Is grarT.matlcally correct. Further more, a stochasl lc auton1aton, that Is learned by experiments, qives the probabi lity of hav ing a ct:rtaln description of lhe pllch contour, given a syntacllc fo r-m. The hypotesis about a sentenc e Is evaluated by the prob al.li li t y o f that sentenc e, given the spectrogram and the pllch contour. If the sentenc e does not match th" spectrogram or the pllch contour, a general los" func ti on i s eva­ luated, proportional to the Information carrhtd by the componef"lta of the .,eniC"!' ce that are not matched, Gvnerallzlng such approach requires 81"1 e v aluat ion o f the Information cnrried by the various elem~nts of a spoken sentence; the pro­ bability of such clements (syllables, word·, phrases) depends on the conte 101 t, The likelihood function and the losa functions are lhe main fac tor& on which the control strategy Is based. Given a certain knowledge abOut a spoken s.enlence ond a set of syllables deteclrd with a proper probabi l i t y, the hypotheses to l.le verified first are those corrttsponding to the maKimum Ilk· llhOod and/or tho minimum luss emong the poaalblltles tht.l are aemet'ltlcally and syntac tically correct. A succei.stul verlflcallon of an hypothesis can alter the &egmentatlon and the set of detected syllables on which the choice of the succes&ive verifi c ation Is baaed, ## Conclusion A project for SUS has been presented. Som· methods have been lnlroduced for evaluating rigorously an hypothesis made on · apectrogram or on a pitch contour. It became evident that a lot of e~pcrlmental work haco to b e dor'\e in learning speclral and prosodic features and In correlating lhem with the str.l-1 cture of the language. A tool for thlt purpose has been provided. A alstematlc application of the propo.,ed algorithm for mf'chan~ cal Inference will help In Investigating how the syllable features can var-y for a a Ingle speaker, how they are eff·cted by the context and In which cases they can be subdivided Into shorter Se9fnenla lndlpendent each other Or when segments larger then syllables have to be considered as units for- speech understanding. In fact the elements of a description, that correspond to prlml tlve spectral forma, have a probability that depends on theo previous ·lemeonta like In a Markov proceaa, but It Ia not known how many ahovld be these Pf"'evl ­ ous elements. Certainly a apeclal purpose hardw..-e able to allow a parallallam of many simple operations will b4l suitable for carry ing enough e H. peorimeonts and reaching lf'l ·cceptable degree or learning. II Ia hoped that the development of speech l.W'tderatandlng syatema will help In conceiving and check models or theories on hum., speech perception, ## Acknowledsements Ttia work waa performed et the Centro Eieborazlone Nurnerele del Sevnall 1 Turin, Italy Wid waa supported by the Consiglio N·z lonale delle Rlcerche of Italy. ## References 1. 1/ Lesser v.R., Fennel R.O.,Erm., L.D . and Reddy D.R.,: "Organize- The le101ical. semantic .,d syntactic levels lion of Hearsay II S~ech Understand ing System", Proc. IEEE Sympo- These levels hove b een conceived In a almple way and will be sium on Speech Recognition, Carnegif-Mellon Un ivers i ty (A.pril 197*). 2. de5crlbed briefly. The protocol used so far was Introduced only for experlmtt!! pp. II ..:. 21 , tal purpotoe and refers to on automated reser-vation and Information •ystem for tra ., ~ls. The vocabulary Is actually of about 200 words and the syntaM and SP.3 f 75>7.