Learning Phonetic Features Using Connectionist Networks

1 Introduction

Connectionist networks offer significant advantages in ad­ dressing problems of machine perception because of their inherently parallel structure, which is well matched to the biological architecture that has served as their paradigm. Their learning capabilities, robust behavior, noise toler­ ance and graceful degradation are all capabilities which are becoming increasingly well understood and documented [11].

The solution of certain perceptual problems requires that the temporal relationships among stimulus character­ istics be properly represented. This is especially true in speech recognition, where the relationship between time and frequency is wonderfully complex. In the production of speech, basic speech units (phonemes) are integrated into a smooth sequence, so that the acoustic boundaries can be very .difficult to specify. Moreover, phonemes are

·Thanks to Alex Waibel for helpful diaeuaaion and analysis.

tThis research was supported in part by DARPA grants N0001485-K-0018 and N00014-85-K-0807, NSF grants DCR-86-07156, DCR8501482, MCS8219196-CER, MCS8207294, 1 ROl-HL-29985-01, U.S. army grants DAA6-29-84-K-0061, DAAB07-84-K-F077, U.S. Air Force grant 82-NM-299, AI Center grant NSF-MCS-83-05221, U.S. Army Research Office grant ARO-DAA29-84-9-0027, Lord Corpor&­ tion , RCA and Digital Equipment Corporation.

Lokendra Shastri

Department of Computer and Information Science University of Pennsylvania t

often co-produced (coarticulated), so that the phonemes exert a strongly context-dependent interaction. Thus, the perception of speech depends on the correct analysis of dynamic temporal/spectral relationships.

The connectionist network approach is attractive be­ cause it offers a computational model which has inherently robust properties. The networks consist of simple process­ ing elements which integrate their inputs and broadcast the results to the units to which they are connected. Thus, the network response to input is the aggregate response of many interconnected units. It is the mutual interaction of many simple components that is the basis for robustness.

The problem of designing connectionist networks which can learn the dynamic spectral/temporal characteristics of speech has not yet been widely studied. Most work in connectionist networks so far has focussed on the static re­ lationship between input/output pairs, such as associative memories [6,4], various encoding, decoding, parity and ad­ dition problems [10], and mapping from word spelling to phoneme labels [11].

Learning to associate static input/output pairs can be accomplished with layered connectionist networks with feed­ forward links alone. Learning pattern sequences requires network state information, which can be provided by feed­ back from the network output to the input [6,5,12,10,9] . The idea of learning pattern sequences has been applied to a speech task using Boltzmann machines [9].

The experiments reported here were designed to ex­ plore the capabilities of parallel networks to learn dynamic propertie& of time-varying data. We choose a standard speech recognition problem to test the extent to which a connectionist network could form an internal represen­ tation of the temporal/spectral characteristics which dis­ tinguish two similar words. A network architecture was selected in which the hidden and output units included self-recurrent links. This approach is distinguished from the pattern sequence approach in that the feedback is in­ ternal to the network and distributed. Thus, the dynamic response of individual units must be learned in solving the discrimination task.

2 Experiment

The discrimination between the minimal pair "no" and "go' is a typical speech recognition problem, which is in­ cluded in a standard database for evaluation of speech rec­ ognizers [1]. The utterances "no" and "go" share for the major and final portion the voiced phoneme /o/. The "no" utterance is characterized by a lower energy nasal murmur preceding the transition to the back vowel /o/. This nasal murmur has a formant structure which is due to the cou­ pled resonances of the closed oral cavity and open nasal cavity. The "go" is distinguished by a very low energy voicing interval during the lingua-palatal closure, a brief burst as the closure is released, and a voiced transition to the full vowel.

The distinction between "no " and "go", therefore, is concentrated in the brief interval of relatively low energy at the beginning of the word. These differences consist in the relative voicing energy, burst spectrum, and formant value and transition pattern.

2.1 Data

The data used for this experimental work consisted of speech data for a single speaker (GD) from Texas Instru­ ments standard isolated word recognition database [1]. The speech data was played into a commercial speech recog­ nition device (Siemens CSE 1200), where it was passed through a 16-channel filter bank, full-wave rectified, log compressed and sampled every 2.5 milliseconds. Twenty­ six repetitions of each word comprise the corpus, for a total of fifty-two utterances (26 "no" and 26 "go"). The filter bank response to the training utterances is shown in Figure 1.

2.2 Network Architecture

For this initial experiment; a three-layer connectionist net­ work consisting of an input layer, one hidden layer and an output layer was implemented, as shown in Figure 2. The sampled speech data flowed through the network in time sequential order. Thus, the 16 channel energies were ap­ plied to 16 input units, from which activation spread to­ ward the output units simultaneously as the input units were updated by sequential speech samples. This design will be referred to as the temporal flow model, or, more simply as the flow model.

Other approaches have used an array of input units, and represented time along one index of the input unit array [8,2,3,7]. In this case, time is spatialized across units. The temporal flow model was chosen because it does not require 'chunking' of variable length utterances onto a fixed size network, it avoids the problem of temporal alignment and symmetry, and the temporal flow model seems to be closer to the biological model of speech processing.

2.2.1 Unit Functions

The functions which define the unit behavior were cho­ sen from ones in common use in connectionist networks [11,10]. The unit outputis a nonlinear (sigmoid) func­ tion of the unit potential, which is a simple weighted sum of the output values of units connected by afferent links. The weights correspond roughly to the effect of synaptic strengths. The sigmoid function has the desirable proper­ ties of a bounded output, non-linear characteristics, and a response threshold. These functions approximate the com­ putational properties of neural cells, and have convenient mathematical properties for the learning algorithm used in this experiment.

For this experiment, an extended form of the back-pro­ pagation learning algorithm was chosen to accommodate networks with recurrent links [10,13).

The error-propagation algorithm modifies the unit con­ nection weights in order to minimize the mean squared error between the actual and desired output values. The weight change rule can be written as:

where O;(t- r) is the error signal at unit j at timet- T, with respect to the target values at the output units at timet [13). This error is given by:

for a~ T.

The error signal for an output unit is defined by the difference between the actual and target values, times the unit function slope at time t:

The value ofT was limited to a small value to limit the recursive computation. The weight changes were made after each time step. These factors introduced approxima­ tions into the computation of the gradient.

2.2.3 Target Function

The target function for the output units used in the no/go discrimination experiment consisted of a simple ramp. For the output unit which corresponded to the utterance be­ ing trained, the ramp increased from a value of 0.5 to 1.00 over the duration of the utterance. The other unit was correspondingly decreased from 0.5 to 0. This represented the intuition that evidence for or against a particular word accumulates over its duration, and reaches a level of con­ fidence after the utterance is completed.

3 Results

The parallel connectionist network experiments were con­ ducted on a sequential machine using a network simulator, written specifically for this experiment. The network de­ scribed previously was trained on a single pair of no/go utterances by a single speaker for 6000 training iterations.

The value of the squared-error term during learning was observed; it was neither monotonic decreasing nor a smooth function of the number of optimization itera­ tions. This is thought to be due to the local nature of the weight change algorithm, and the limited extent of back­ propagation in time. The error value did reach sharply­ defined minimum value after 4000 iterations; the network at that point was chosen for further study.

3.1 Output Unit Response

The response of the output units for the network at the selected critical point in the learning process was recorded, and can be seen in Figure 3. The output units respond in equal and opposite ways to the input stimuli; in addition, their time response roughly approximates a ramp. Since the learned response closely fits the training function, the network exhibits correct discrimination between the pair of items in the training set.

The significance of this result should not be overlooked. First, the local application of a global optimization metric provided a successful path to the desired network response pattern. Second, although no segmentation decisions were made, the network was able to form a discriminating spec­ tJ;al feature which was localized in time. Third, the approx­ imations of constant weight value, and restrictions to max­ imum T value in the extended back-propagation algorithm did not prevent convergence to a good solution. Fourth, although the shape of the error contour is unknovtn, it is al­ most certainly not smooth; consequently, the learning path apparently avoided local minima in arriving at a solution.

3.2 Extension to Test Set

In order to test the generality and robustness of the inter­ nal representations obtained from the training word pair, the network of least squared error value was tested on a set of 25 additional pairs of no/go utterances by the same speaker. Using a simple deterministic decision algorithm, the input word could be clearly categorized by the network response. Under these conditions, the trained network suc­ cessfully discriminated all but one of the test cases (98% accuracy).

The responses of the hidden units were analyzed for the 50 test utterances as well as the 2 training utterances. In nearly every respect, the hidden unit responses of the test utterances were isomorphic to the response to the training data. A single hidden unit provided the discriminatory re­ sponse. In the single error case, this unit failed to respond to the input data. The energy levels for this utterance were very low, especially in the mid to upper channels.

4 Discussion

Although the results of this initial experiment are unex­ pectedly encouraging, there are several problems which need to be addressed. The stability of the learning algo­ rithm needs to be improved. This could be accomplished through better target functions, greater accuracy in com­ puting the gradient, or improved learning algorithms. These ideas for improvement have been addressed in subsequent work. More powerful optimization algorithms (second­ order iterative methods) have have resulted in stable learn­ ing and greatly increased learning speed.

5 Conclusions

In conclusion, several interesting results emerge from this experiment. Using a connectionist network with a tempo­ ral data flow architecture with recurrent links, and using an coarse approximation of the desired output as a teach­ ing function, a successful discriminatory mechanism was learned. This discriminatory feature was formed without segmentation and without a direct comparison of the two items.

The discriminatory mechanism turned out to be very robust , even though based on a single training sample. This result is very encouraging for further research with connectionist networks in deriving robust discriminatory features of phonetic classes.

Obviously, the goal of this research is to structure net­ works which can learn the complete set of phonetic class discriminations, so that it could support real-time, con­ tinuous speech recognition. This requires larger networks, which for efficiency, may need to be partitioned and re­ combined. Initial steps in this direction. have been taken by training networks to discriminate the stop consonants in CV words using various vowels.