Speaker-Invariant Features for Automatic Speech Recognition

1 Introduction

Inter-speaker variation is a major cause for the degradation in the performance of Automatic Speech Recognition (ASR). This is especially important in speaker-independent (SI) ASR systems which are typically built to handle speech from any arbitrary unknown speaker, e.g. in applications such as directory assistance. As a rule of thumb speaker-dependent (SD) ASR systems have half the error rates when compared to SI systems for the same task. This difference in performance suggests that the performance of SI systems can be significantly improved if we can account for inter-speaker variations. Note that in practice, we cannot directly build SD systems for large vocabulary speech recognition tasks, since the speech data required from that speaker will be too enormous even for a co-operative user. Instead, we build SI systems and then "adapt" the system for a particular speaker to make it speaker-dependent. Hence, irrespective of the application, there is a lot of interest in improving the performance of SI systems by accounting for inter-speaker variability.

In general, two speakers enunciating the same sound have very different pressure waveforms and the resulting spectra are very different. Fig. 1 shows the smoothed spectra (after smoothing out pitch) for two speakers enunciating the vowel /aa/. As seen from the figure the two spectra are very different even though it is the same vowel spoken by two speakers. Since ASR systems use features extracted from the spectra, the features are themselves different for the same sound enunciated by two speakers. Therefore for an automated system to recognize these different features as belonging to the same sound is difficult leading to a degradation in performance. Note that there will be a certain amount of variability for the same sound spoken by the same person, i.e. intraspeaker variability, which is handled by the statistical model. However, in general inter-speaker variability is substantially larger than intra-speaker variability, leading to "coarse" statistical models and hence increased confusion between the sound classes.

There is a lot of research that is being done in trying to understand the mathematical relationship between spectra of two speakers enunciating the same sound. This has important implications in other areas apart from speech recognition, such as in vowel perception, hearing-aid design and speech pathology. One of the major causes for inter-speaker variations is attributed to the physiological differences in the vocal-tracts of the speakers. Based on this idea, in ASR a commonly used model to describe the relation between spectra of two speakers enunciating the same sound is given by

where SA(f) and SB(f) are the spectra of speakers A and B respectively; and α denotes the uniform (constant) frequencywarp factor. We refer to the above model as linear-scaling model, since the two spectra are just scaled versions of each other. The motivation for linear scaling comes from the fact that to a first-order approximation, the vocal tract shape can be assumed to be a tube of uniform cross-section and for this simplifying approximation, the resonant frequencies (which characterize different sounds) are inversely proportional to vocal-tract length ([Wakita, 1977]). Therefore, differences in vocal-tract lengths manifest in scaling of resonant frequencies with the scaling being inversely proportional to the vocal-tract length.

In practical ASR systems, we have access only to the acoustic data from the speaker and therefore do not have any idea of the speaker's vocal-tract length. Therefore, we do a brute-force search for the optimal scaling-factor, α by trying out different values of α. The optimality criterion is based on maximizing the likelihood with respect to the SI Hidden Markov Model (HMM). Since in ASR we use features derived from the spectra, for each value of α we scale the frequency-spectra appropriately and then compute the features. Therefore for each value of α, we recompute the feature of the ith utterance, Xαi , for that scale-factor. We then find the optimal scale-factor by

where α represents scale factor. λ is the HMM model and Wi is the transcription of the utterance. Due to physiological constraints of the human vocal-tract apparatus, α lies in the range of 0.80 to 1.20. Therefore, in state-of-the-art ASR systems, we compute the features for different values of α from 0.80 to 1.20 in steps of 0.02 and find the optimal α using Eq. 2. This is a computationally expensive process since for every frame the features have to be computed 21 times corresponding to different α, before finding the optimal feature for that frame after a maximum-likelihood (ML) search. Hence, a lot of research is being done to have alternate normalization schemes that are computationally efficient.

Recently, an alternate method for generating features that are invariant to speaker-variations due to the linear scaling (see Eq. 1) was proposed [Umesh et al., 1999]. In this method, the features are insensitive to the scale-factor α and hence there is no need to search for the optimal α. We describe the method in the next section.

2 Review of Scale-Invariant Features using Scale-Transform

In the linear-scaling model of Eq. 1, the speaker-dependent warp-factor, α is a multiplicative constant and is independent STCC

Figure 2: Block diagram for the computation of STCC

of frequency. For such a model, log-warping the frequencyaxis of the spectra of speakers separates the linear scaling factor as a translation factor in log-warped spectral domain, i.e.,

Hence, in the log-warped domain, λ, the speaker-dependent scale-factor separates out as a translation factor lnα for the linear scaling model of Eq. 1.

Let the Fourier-transform of the log-warped spectra sa(λ) be denoted as Da(c) and that of sb(λ) be denoted as Db(c), i.e.

Using the property of the Fourier-transform that a translation in one domain appears only as linear-phase in the other domain we have,

Therefore, the speaker-dependent term α appears only in the phase term. Hence the magnitudes of Da(c) and Db(c) are identical and independent of the speaker-variations i.e.

We can therefore use the magnitude as speaker-invariant features for ASR.

We can show that the above procedure is equivalent to taking the Scale-Transform [Cohen, 1993] (a special case of Mellin transform) of SA(f) and SB(f) and then using the magnitude of Scale-Transform as features in ASR. In this paper, we refer to these features as Scale-Transform Cepstral Coefficients (STCC). The steps to compute STCC are shown in Fig. 2.

Fig. 3 illustrates the idea of using Scale-Transform to obtain speaker-invariant features for the linear-scaling model of Eq.1 using a synthetic example. Fig. 3(a) shows synthetic spectra of two speakers enunciating the same sound that are exactly scaled versions of each other corresponding to the linear-scaling model of Eq.1. In Fig. 3(b) we show the same spectra in the frequency-warped(i.e. log-warped) domain. As seen from the figure, the frequency-warped spectra are exactly shifted versions of each other as expected from Eq. 3. Finally, we take the Fourier-transform of these frequencywarped spectra and then take the magnitude of the Fouriertransform to get the STCC features. These are shown in Fig. 3(c). As seen from the figure, the STCC features are identical, and are therefore invariant to speaker variations that are present in the original spectra as seen in Fig. 3(a).

Since the linear scaling model of Eq. 1 is only a crude approximation, recently [Umesh et al., 2002] have proposed the use of mel-scale for frequency-warping and not the log-scale. With this mel-scale frequency-warping they claim better results [Sinha and Umesh, 2002]. Therefore, for the purposes of this paper, we will use mel-scale warping to compute the STCC features.

We now compare the performance of conventional MFCC features with STCC features. TIMIT database was used for the experiments and we considered the classification of 42 monophones. The train and test set consist of 155015 and 56424 monophone occurrences for adult male and female speakers respectively. The HMM models consisted of 3 emitting and 2 non-emitting states, with left-to-right and without skips over states. Two mixture Gaussians with full covariance matrices per state were used. The features vectors are of 39 dimensions comprising normalized log-energy, c1. . . c12(excluding c0) and their first and second order derivatives. Conventional MFCC features are computed as described in [Lee and Rose, 1998] while the STCC features are computed as described in [Umesh et al., 1999] and a block diagram is shown in Fig. 2.

The classification performance between STCC and the MFCC features are shown in Table 1. As seen from the Table, STCC features have a lower classification performance when compared to MFCC features. Since STCC provides speakerinvariant features, it should have provided improved normalization performance when compared to MFCC. The reason for the degradation is the complete loss of phase-information. We discuss about the loss of phase information in Section 3 and propose new features to overcome this loss of information by introducing an average phase vector which is discussed in detail in Section 4.

3 Drawback of STCC - loss of phase information

Theoretically, STCC provides speaker-normalization by exploiting the fact that the magnitude of STCC are identical as seen in Eq. 6. However, note that Da(c) and Db(c) are complex quantities with associated magnitude and phase. Let

Therefore |Da(c)| = |Db(c)|. However, we have completely lost the phase information ejφ(c) which is also important for discrimination of phonemes. This loss in phase information degrades the performance of the STCC and even though it is insensitive to speaker-variations, the final performance is inferior to the MFCC. In the next section, we propose a method to overcome this problem and help improve the performance of the STCC.

4 Proposed Improvement over STCC features

As seen in the previous section, the loss in phase information ejφ(c) leads to degradation in performance when compared to MFCC features even though STCC features are invariant to speakers as seen in Eq. 6. As seen in Eq. 8, Da(c), Db(c) contain both the magnitude and phase information and the speaker-specific factor, α, appears only in the phase term. Hence retaining the complete phase will result in no normalization since the lnα will also be present. On the other hand, taking only the magnitude will result in the loss of ejφ(c) information which provides additional information for discrimination between vowels.

In this section, we propose a method to incorporate the phase information ejφ(c) but not the speaker-specific lnα in the STCC features. The basic idea of this approach is to estimate an "average" phase vector for each phoneme using training data from all speakers.

As previously discussed, for the linear-scaling model, the speaker differences manifest themselves as speaker-specific shifts in the log-warped domain as seen in Fig. 3. The STCC exploits this fact by considering the magnitude of the subsequent Fourier-transform but in the process loses the phase information completely. Our approach is to estimate the average phase for each phoneme from the training data and use this same phase-vector for every occurrence of that phoneme irrespective of the speaker. For example, for the phoneme /ae/, our average-phase STCC features with acronym APSTCC will be of the type |Da(c)|ejφavg ae(c) . We will first illustrate this idea through a synthetic example that is discussed below.

Consider that a particular phoneme is enunciated by three different speakers and the corresponding spectra after frequency-warping are shifted versions of one another as shown in Fig. 4(a). The three spectra in the Fourier-domain will differ only in a linear-phase term and this can be seen mathematically as:

If we now compute the average phase of all the three phases, i.e.

then the average-phase φavg(c) is same as the phase of the phoneme, φ(c), with an additional linear-phase term that corresponds to the average of all the three shifts. Therefore, the average phase φavg(c) preserves the phase of the phoneme with an additional term corresponding to the average shift of the spectra of the training speakers. If we now use this average phase for all speakers along with the magnitude of the STCC feature, we get speaker-normalized features. This can be easily seen by recalling the fact that |Da(c)| is constant for all speakers; and when it is multiplied by the same phase term ejφavg(c) for all speakers, then the resulting features are same for all speakers. This is shown in Fig. 4.

In practice, since the processing is done in the discretedomain, the phases are 2π wrapped, and hence we first unwrap the phases before averaging. The other important practical limitation at this point is the fact that we need to know the phoneme a priori so that we can add the appropriate averagephase of that phone to the magnitude vector |Da(c)|. We are now working on practical algorithms to find the appropriate phase for a given frame without having the knowledge of which phoneme the frame came from. However, as a first step, we will assume knowledge of appropriate phase vector and test the efficacy of this method of normalization and compare it with the conventional method of normalization.

5 Performance of the Proposed Normalization Method

In this section, we will compare the performance of the proposed normalization scheme with conventional normalization scheme.

Here we considered the classification of 8 "most confusable" vowels in contrast to the classification of 42 monophones used for MFCC and STCC feature vectors discussed in section.2. The main reason was to study the effect of our proposed phase-estimation procedure in more detail. The vowels were extracted from TIMIT database, where the train and test set consisted of 22686 and 8265 utterances respectively. The HMM models consisted of 3 emitting and 2 non-emitting states, with left-to-right and without skips over states. We used single mixture Gaussian with diagonal variance for each state. The feature vectors are of 26 dimensions comprising normalized log-energy, c1. . . c12(excluding c0) and their first order derivatives.

The conventional normalization scheme is based on Eq. 2, which involves ML estimation of the warp-factor α. This method is also popularly known as Vocal-tract length normalization (VTLN). As discussed in the introduction, we do a brute-force search for the optimal α by computing the MFCC feature for each α by appropriately scaling the mel filter-bank and choosing the feature that maximizes the likelihood with respect to the statistical HMMmodel and the given transcription Wi. Note that although the method needs transcription, there is graceful degradation when there are errors in the transcription.

In our proposed method, we are still working on methods to find the optimal phase-vector to multiply a given frame of speech. As seen from Table 2 if we are given exact transcription, then we can multiply by the correct average-phase vectors and the performance is exceptional. On the other hand, even with exact transcription the VTLN method is far inferior to the proposed method of using AP-STCC features. But, if we use the recognition output of the baseline recognizer (i.e. without normalization) as the transcription then the VTLN normalization method degrades gracefully while the AP-STCC completely falls apart - it is worse than even baseline. Note that we do not need to know the transcription for AP-STCC, but a method of finding "optimal" phase-average vector for each utterance. We are working on various distance measures to obtain an "optimal" solution. However, it is important to note that at least under ideal conditions (when true transcription is known) our method shows that it is possible to remove almost all the speaker-variability which conventional VTLN-MFCC is never able to achieve.

Another approach to measure the normalization performance is to measure the separability of the vowel models using the various normalization schemes. One good measure of separability is the F-ratio between models considered pair-wise. Tables 3,4,5,6 shows the F-ratio for the unnormalized features, the VTLN features and proposed APSTCC features. These tables show F-ratio of models that are built using training data for which true transcription is always known. The F-ratio shows the separability between models and the higher the number the better. Note that since MFCC and STCC are computed slightly differently the features are slightly different and hence there are small differences in performance in the un-normalized case. From the Table, it can be seen that the separability between the vowels models are excellent for the AP-STCC features.

6 Conclusion &Discussion

In this paper, we have proposed a method of obtaining speaker-invariant features for automatic speech recognition. Our method is motivated by the fact that conventional methods reduce inter-speaker variability by doing a brute-force search for optimal features and are therefore computationally very expensive. Recently a method has been proposed that uses a special transform called ScaleTransform to obtain speaker-invariant features. Since the Scale-Transform uses only the magnitude of the features to obtain speaker-invariance, there is a loss of discriminability between phonemes due to the loss of phase information. In our proposed method, we use the average-phase of the corresponding phoneme in place of the lost phase information of STCC. We are currently working on various distance measures to find the appropriate average-phase for normalization. Using vowel classification experiments and F-ratio measures we show that if we have optimal average-phase information then the proposed method provides excellent normalization performance when compared to the conventional VTLN method.

7 Acknowledgments

This work was supported in part by funding from the Dept. of Science & Technology, Government of India under project No. SR/S3/EECE/0008/2006-SERC-Engg.