Use of Procedural Knowledge for Automatic Speech Recognition

1. INTRODUCTION

Recent results on Automatic Speech Recognition (ASR) and Speech Analysis suggest that progress in designing recognition devices and in advancing speech science knowledge may arise from an integration of the so called cognitive and information-theoretic approaches [1].

The cognitive approach attempts to infer analytic knowledge about possible speech invariants and their relations. Work by Zue [2], Klatt [3], Stevens [4] and De Mori et al. [5,6] are along this line.

The information theoretic approach is based on a performance model containing states and transitions between any pair of states [7]. Probabilities can be learned that the system is in any of the model states or is changing state through any of the allowed transitions. Furthermore, the model generates in each state or in each transition, observable system parameters or descriptors according to some statistical

distribution.

This paper proposes an attempt to integrate the two above mentioned approaches.

The idea is that of extracting speech properties using knowledge about acoustic correlates of linguistic units. For an abstract linguistic unit, the corresponding acoustic correlates have attributes which may differ from an instantiation to another due to the fact that different speakers produce different signals even if they intend to pronounce the same sound. Attribute statistics of sound properties collected on a large variety of pronounciations of the same sound from different speakers are probably the best knowledge we styles. Furthermore, some expected acoustic properties can be missed in some cases and some unexpected properties can be detected in some other cases. These aspects can also be characterized by stochastic performance models.

An important problem ansmg when large vocabularies have to be recognized is that of identifying a possibly small set of Sp.e.e..ch l.lniLs (.Sll) with which all the possible words and word concatenations can be obtained by compilation. The learning problem is then reconducted to the conception of a performance model for each SU.

2. A MODEL FOR COMPUTER PERCEPTION OF SPEECH

The speech signal x 1 (t) is generated by a discrete and finite sequence of actions

where ak ( tk ) denotes an action ending at time tk ;a 1 ( t d represents the silence preceding the beginning of a sentence. ·

is applied which produces A . The relation R 1 may depend on the speaker, his/her mood, state of health and history. As R 1 may produce several As for the same S, probability distributions for all the possible As can be derived using a generative model.

The speech signal x 1 ( t ) is generated by the sequence of actions A using another relation

R 2 depends on the anatomy of the speaker. Again, the same actions may produce different signals, because the speech production system is soft and its behavior is affected to some extent by the environment.

If the speaker does not read but generates a sentence from a set C of concepts, then a third relation is applied:

R 3 may depend on the speaker and his/her culture. Statistical models can also be used for characterizing

3. PROCEDURAL NETWORKS

A Procedural Network (PN) can be described with a formalism similar to that used for an Augmented Transition Network Grammar (A TNG ). This formalism has been successfully used for Natural Language and Pattern Recognition [8]. A PN is a 5tuple

where j is the network identifier, Q is a finite set of states, A is a finite set of directed arcs, q 0 EQ is the initial state and q 1 is the final state. Without any loss of generality we consider only PNs with a single initial state and a single final state.

Each arc a; EA is a 5 - tuple:

where qbi EQ is the starting state ~f a; ,qei EQ is the terminal state of a; ,P; is a measure associated to the arc (it can be a weight or a probability according to the scoring method used by the PN supervisor described later on), condition; is a condition and action; is an action; both of them are associated to the arc. The conditions can be categorized in two classes:

COND n

refers to a user defined condition n.

DEFAULT r

refers to a default condition (it is satisfied only if no other condition of any arc whose starting state is q6; returns a measure greater than r).

The actions are executed by the PN supervisor and can be categorized in five classes

EXEn

executes a user defined action; such an action is usually a "matcher" which performs some computations on the input data and returns a result.

PUSHi

is defined as follows. Let's assume that PNi has an arc that contains PUSH i. Let 71' j be the process that executes PNi. When the arc is reached whose associated action is PUSH i, the execution of 71' i is suspended. The state of 71' i is pushed on the top of the stack of the PN supervisor. A new process 71'; that executes PN; is created and executed. When the final state of PN; is reached, the last arc of PN; is considered. It has associated either a POP ABS f or a POPCOND f action. This action is executed. It returns scores computed by PN;. These scores are passed to 11'j whose execution is resumed while 71'; terminates.

POPABSf

is associated to the final state of a PN. It stops the execution of the current network process and the result of the execution of the user defined function f is returned.

POPCOND f

This action is also associated to the final state of a PN. It has a synchronization capability that stops the execution of the current network if all the paths in the network leading to the final state have propagated their contribution to the computation of the scores the PN has to provide. If the condition is reached, then the result of the execution of the user defined function f is returned.

JMP

makes the score associated to qbi propagate to qei without any change.

Each PN is associated a Working Memory (WM). Actions associated with the arcs of a subnetwork produce descriptions stored into the subnetwork WM. When a push to a subnetwork is made, the network supervisor may link the subnetwork WM with other WM, thus establishing the viewpoint within which conditions are tested. Most of the actions associated with arcs include plans, Hidden-Markov-Models (HMM), local parsers, rule-based inference units. All these tools are used for extracting an unambiguous description D of a speech pattern and for computing an a-priori probability for an hypothesis H:

The PN supervisor keeps up to date a search space where each node is represented by the following four-tuple:

where:

-q is a state of a PN, with a buffer containing the information propagated by the actions executed before reaching it,

-"context" is the context (viewpoint) in which the conditions and actions of the arcs starting at q have to be executed,

-T is the starting time of the speech signal for the execution of sensory procedures invoked by the actions associated with the arcs starting at q,

-"score" is the score of the hypothesis contained in or implied by "context" up to T. Composite scores can be evaluated as likelihoods:

where Pr (H) is obtained by a language model.

The size of the search space can be kept small in spite of a large number of states in the PN if conditions and actions are properly chosen and placed in the network.

4. AN EXAMPLE OF APPLICATION

Multi-speaker recognition of isolated letters and digits was performed on four (two male and two female) speakers.

The vocabulary used for this experiment is defined in Table I. The speech utterance is segmented into Acoustic Segments (AS) with an algorithm described in [5].

this relation.

The generation of x 1 ( t ) can be seen as the application of the following composite relation:

according to the scheme shown in Figure 1.

Recognition consists in applying the relations in the opposite direction. Unfortunately we have only a limited knowledge of these relations. We have used it for building speech synthesizers. We do not even know the alphabet :EA { ak } for the elements of A, although we know alphabets :Ec and :Es for the elements of C and S respectively. Furthermore, signal x 1(t) is affected by noise and is transformed into x ( t ) through the acoustic channel.

As we do not know :EA , nor we know R 2, we can characterize actions by descript.ions of what they produce. According to this approach, the perception of x ( t ) consists in extracting a sequence of descriptions:

where d 1 (r 1 ) describes the silence preceeding the beginning of the speech signal and d; (r;) describes the segment of x ( t ) between the time instants T; -I and T;_

Segments of D can be 10 msecs. frames or intervals of variable duration obtained by a segmentation algorithm like the one proposed in [5].

The descriptions D can be obtained by perceptual .a.c1J..ans in analogy with the generative scheme. Perceptual actions, as well as generative actions have to be defined and used according to a criterion of economy. That is, there must be a limited number of actions (operators) based on which a variety of networks of actions, can be built.

Recognition can be seen as a combination of a relation

that is the perceptual counterpart of relation R 1 used for speech generation, and a relation:

that is the perceptual counterpart of R 2 (A ,x ,( t )).

The relation L 2 (x (t ),D) is deterministic in the sense that it can produce only one description D for a signal x (t ). Description D can be of fixed duration , i.e. a descriptive phrase is generated at constant time intervals, or of variable dmation , i.e. descriptions can be generated for intervals of different length. If we want to maintain the analogy with the production model just outlined, D should be of variable duration because the articulatory actions (gestures) are of variable duration.

Descriptions must refer to parameters, morphologies and properties that are characteristic for a sound and exhibit low variances when many speakers, different microphones and environments are considered.

In practice, fixed duration models have been developed and tested with a considerable degree of success mostly in speaker-dependent systems. In one 'of the most successful systems developed so far [7], D is a sequence of symbols obtained every 10 msecs. by vector-quantization with a process that is speaker­ dependent and context-independent.

Relation L 1 (D ,S ) has to capture two different types of knowledge. The first type of knowledge is a relation:

between a sequence U of Sp.ee.ch llnit.s (SU) and corresponding descriptions D . There are speech units like the plosive sound I b I for which a large variety of different descriptions · D are perceived as the same. :!Olln.d. Relation L 11 is many-to-one and it could be interesting to collect statistics of the elements of the universe of acoustic descriptions that produce the perception of the same linguistic sound. These statistics may represent distributions of acoustic patterns produced by a single or many speakers having the intention of producing the same sound. Statistics may also take into account characteristics of background noise.

A second type of knowledge is a relation:

L dU ,S)

(10)

where S is a linguistic entity like a sentence and U is a sequence of Speech Units. L 12 can also contain statistics.

L 12 may represent how different speakers may have different pronounciations of the same. word. A stochastic model representing a word W in terms of SUs can be built.

An interesting possibility, we would like to explore h.t this paper, is that of designing L 2 and L 1 procedurally , through actions to be performed on x ( t ) in order to obtain D , U and S .

It seems that yariahle .d.epj.h descriptions can be very useful in complex tasks where a preliminary selection of hypotheses has to be done based on robust but simple descriptions and then a more detailed analysis has to be performed involving levels of depth depending on the competing hypotheses, or just on acoustic preconditions.

The entire perception model can be represented by procedural networks which · invoke subnetworks at several levels. At each level, different types of units can be defined and statistics of their components can be collected.

Each AS is processed in order to hypothesize the Speech Units contained in it. For this purpose, actions are introduced for describing the AS head, its vocalic part, and its tail according to the PN shown in Figure 2.

Speech Units correspond in this example to phonemes and are characterized by "place and manner of articulation".

The AS head is analyzed by attached procedures (actions) performing an Elaboration-Decision (ED) paradigm. There are two possible EO-actions for the head of an AS, namely:

• -plosive head
• -fricative (including afficate) head

The choice of the ED action · is made by disjoint conditions associated to arcs. These conditions are regular expressions of Primary Acoustic Cues introduced in [5]. State M1 in Fig. 2. will be reached by only one arc. Using techniques partially described elsewhere [6], hypotheses about the place and manner of articulation for the speech unit in the head subsegment are generated and scored by the following a-priori probability:

After state M1, the ED-action "vocalic" is executed. It segments the vocalic part of AS into stationary and transient units.

Let be such subsegments. For each segment tlz spectral lines are considered as data (see [Q] for details) and a­ priori probabilities about place and manner of articulation are obtained by HMMs of spectral lines in the segment tis . For each subsegment and for each consistent "place-manner" pair, the following probability is computed.

From state M2 to state M3, EO-actions for the tail of AS are executed similar to those used for the head. A probability

scores the hypotheses of the tail subsegment. The data extracted in the head, the subsegments of the vocalic part and the tail -can be assumed to be independent. The "select" action associated to the POP ABS arc computes for each candidate hypothesis the probability Pr (data /hyp) by multiplying the probabilities of the phonemes which appear in "hyp" :

Preliminary results have been. obtained on a test set of 400 utterances of words belonging to the alphabet defined in Table I and pronounced by four speakers. The voices of these speakers were used for learning only head and tail statistics. An overall recoginition rate superior to QO% was achieved.

ACKNOWLEDGEMENTS

This work was supported by the National Science and Engineering Research Council of Canada under Grant A243Q. We would like to thank Lorraine Harper for her help in the final preparation and layout of the paper.