Understanding Three Simultaneous Speeches

1 Introduction

Recently emerges a new research on understanding arbitrary sound mixtures including non-speech sou~ds and music. Their understanding represents a challengmg and little-studied area of artificial intelligence, automatic speech recognition/understanding, and signal process­ ing. This interdisciplinary research area is called com­ putational auditory scene analysis (hereafter, CASA):

At a crowded party, one can attend. one conversatw.n and then switch to another one. This phenomenon IS known as the cocktail party effect [Cherry, 1953]. As seen in the cocktail-party effect, humans have the ability to selectively attend to sound from a particular source, even when it is mixed with other sounds. Current automatic speech recognition systems can understand clean speech well in relatively noiseless laboratory environments, but break down in more realistic, noisier environments.

Computers also need to be able to decide which .parts of a mixed acoustic signal are relevant to a particular purpose - which part should be interpreted as speech, for example, and which should be interpreted as a door closing, an air conditioner humming, or another per~on interrupting. CASA focuses on the computer modelmg and implementation for the understanding of acoustic events The research topics concerning CASA include mod­ eling, signal processing, sound representational, control and system architecture, and applications as well as sen­ sor integration. Some of these topics were discussed at the IJCAI-95 workshop .on Computational Auditory Scene Analysis [Rosenthal and Okuno, 1997].

At the AAAI-96, the panel entitled "Challenge Prob­ lems for Artificial Intelligence", Brooks proposed two problems concerning sounds [Selman et a/., 1996]:

• Challenge 1: Speech understanding systems that are bas~d on different principles other than hidden Markov models.
• Challenge 2: Noise understanding systems.

Although CASA shares the above interests, its ulti­ mate goals go further; understanding general acoustic signals such as voiced speech, music and/or other sounds from real-world environments.

We propose the problem of Understanding Three Si­ multaneous Speeches 1 (hereafter, the challenge) as a challenge problem for artificial intelligence, in particu­ lar, for CASA. A computer capable of listening to sev­ eral things simultaneously is called Prince Shotoku Com­ puter after the Japanese legendary that Prince Shotoku (A.D. 574-622) could listen to ten people's petitions si­ multaneously [Okuno et a/. , 1995] . Since psychoacoustic studies have recently showed that humans cannot listen to more than two things simultaneously [Kashino and Hirahara, 1996], CASA research would make computer audition more powerful than human audition, similar to the relationship of an airplane's ability to that of a bird.

1 The selection of the word "simultaneous" or "concurrent'' is controversial. The former carries more physical senses, while the latter carries more mental senses; e.g., "separation of simultaneous talkers", "simultaneous voices separation", and "separation of concurrent sentences" make sense. We adopt "simultaneous" because the proposed challenge prob­ lem won't pursuit understanding what each speaker talks about. Understanding what a speaker says without speech recognition, for example, is beyond our problem.

The rest of this paper is organized as follows: Sec­ tion 2 explains the research issues for the challenge, in particular, its relevance and significance to AI. Section 3 presents its feasibility by showing the result of pre­ liminary experiments on understanding two simultane­ ous speeches with/without interfering sounds. Section 4 discusses the detailed plan of our challenge problem. Concluding remarks are given in Section 5.

2 Research Issues for Understanding Three Simultaneous Speeches

In this section, we explain the reasons why we fo­ cus on three simultaneous speeches, not two simulta­ neous speeches and how significant the challenge is to AI researches. We also discuss several research issues involved in realizing understanding three simultaneous speeches. The main research areas related to the chal­ lenge are automatic speech recognition (ASR), signal processing, speech understanding, computational audi­ tory scene analysis, and psychoacoustics.

2.1 Automatic Speech Recognition (ASR)

At present, one of the hottest topics of ASR research is how to make ASR systems more robust so that they can perform well outside laboratory conditions [Hansen et al., 1994]. Conventional approaches for robust ASR are speech enhancement and many techniques for speech enhancement such as noise reduction and speaker adap­ tation have been developed [Hansen et al., 1994; Minami and Furui, 1995] .

One possible approach is to enhance a speech by em­ ploying noise reduction techniques. Once a speech is enhanced, it can be subtracted from a mixture of sounds in waveform. By repeating this procedure to the residue (remaining sounds), it seems possible to extract most speeches from a mixture of sounds.

This approach, however, works only up to two si­ multaneous speeches. The reason is as follows; Most conventional noise reduction techniques assume that the signal-to-noise ratio (SNR) of speech is 0 dB or better. The SNR of speech in a mixture of two simultaneous speeches is approximately 0 dB and thus noise reduction techniques can be applied to two simultaneous speeches. However, new techniques need to be developed for un­ derstanding three simultaneous speeches.

2.2 Signal Processing

Speech separation is more aggressive approach than noise reduction. Adaptive filters are used for speech separation [Ramalingam, 1994] . Spatial information on the sound source plays an important role in separat­ ing a speech from a mixture of sounds. This mech­ anism is called localization, which is performed by us­ ing a dummy head microphone (called binaural sounds) (Blauert, 1983; Bodden, 1993] or by using microphone ar­ rays [Hansen et al., 1994; Stadler and Rabinowitz , 1993] .

For a pair of microphones, localization can be obtained better from binaural sounds than from stereo sounds.

Adaptive window technique for localizing two simul­ taneous voices by using two microphones is also devel­ oped for speech enhancement in real-time [Banks, 1993] . Procedures for enhancing the intelligibility of a target speaker (talker) in the presence of a simultaneous talker is developed by using harmonic selection and cepstral filtering [Stubbs and Summerfield, 1991]. Classification tasks within an automated two-speech separation system are performed by neural net [Roger et al., 1989].

Most of these systems can separate a speech from a mixture of two simultaneous speeches. A speech sepa­ ration system is developed by using harmonic structure and directional information and can extract one speech from a mixture of more than two overlapping speeches [Luo and Denbigh, 1994].

Since the spectrum of speech separated by speech sep­ aration techniques developed so far is distorted, they cannot be applied continuously to the remaining signals. We need to develop new techniques for the challenge. In addition, a segregated speech cannot be used as a input to automatic speech recognition systems due to spectral distortion. We also need to develop an interfacing tech­ nique between speech separation and ASR.

2.3 Computational Auditory Scene Analysis (CASA)

Speech enhancement technologies developed so far fo­ cus on only one speech and treat other speeches or sounds as noise. CASA takes an opposite approach. First, it deals with the problems of handling mixture of sounds to develop methods and technologies. Then it applies these to develop ASR systems that work in a real­ world environment. The main research topic of CASA is sound stream segregation, a process that segregates sound streams that have consistent acoustic attributes from a mixture of sounds.

In extracting acoustic attributes, some systems as­ sume the humans auditory model of primary process­ ing and simulate the processing of cocklear mechanism [Brown, 1992; Slaney et al., 1994]. Brown and Cooke designed and implemented the system that builds vari­ ous auditory maps for input sounds and integrates them to segregate speech from input sounds [Brown, 1992; Brown and Cooke, 1992] . An auditory map represents acoustic attributes such as onset, offset, AM and FM modulations, and formants. Since the integration pro­ cess becomes complicated when treating a mixture of sounds under the real-world environments, the black­ board architecture is used to simplify this integration process [Cooke et al., 1993].

To design a more flexible and expandable system, con­ trol mechanisms are needed. IPUS (Integrated Process­ ing and Understanding Signals) [Lesser et al., 1993] in­ tegrates signal processing and signal interpretation into the blackboard system. IPUS has various interpretation knowledge sources which understand actual sounds such as hair driers, footsteps, telephone rings, fire alarms, and waterfalls [Nawab, and Lesser, 1992] .

Nakatani et a/. took a multi-agent approach to sound stream segregation which extracts individual sound stream from a mixture of sounds by agents each of which traces harmonic structure with directional information [Nakatani et a/., 1994]. They use the Fourier transfor­ mation instead of the auditory model because the former is easy to implement and its properties are well analyzed.

2.4 Psychoacoustics

Psychoacoustic people have studied the human audi­ tory mechanism extensively as auditory scene analysis [Bregman, 1990], but computer modeling has not been exploited yet. Emerging computational auditory scene analysis research focuses on computer modeling and has prompted interdisciplinary studies with psychoacoustic and AI and signal processing communities. In addition, our challenge problem has fostered psychoacoustic stud­ ies on how many simultaneous speeches human can lis­ ten to. Kashino et a/ claimed that human could listen to at most two things simultaneously by performing vari­ ous experiments [Kashino and Hirahara, 1996]. If this is true, the challenge will attempt to make computer audi­ tion superior to human 's capability of listening.

3 Preliminary Experiments

In this section, we demonstrate the feasibility of the challenge by describing the preliminary experiments on understanding two simultaneous speeches (up-to-date in­ formation of our AAAI-96 paper [Okuno et a/., 1996]). This problem is attacked by speech stream segregation, one of the main research topics of computational au­ ditory scene analysis. The whole system consists of two components, speech stream segregation and speech recognition, as is shown in Figure 1.

First speech streams are extracted from a mixture of speeches, and then each speech stream is recognized by conventional automatic speech recognition system.

3.1 Speech Stream Segregation

Human voice consists of harmonic sounds such as vowel and voiced consonants, and non-harmonic sounds such as unvoiced consonants. By assuming the structure of "Vowel (V) +Consonant (C)+ Vowel (V)" of speech, speech stream segregation is realized by the following two subprocesses:

• ( 1) extracting and grouping harmonic stream frag­ ments (harmonic structure extraction), and
• restoring non-harmonic parts by residue (residue substitution).

Rough flow of the computation is depicted in Figure 2. Harmonic structures are extracted from a binaural in­ put by the Bi-HBSS (Binaural Harmonics-Based Stream Segregation) system [Nakatani et a/., 1995; Nakatani et a/., 1996] . Bi-HBSS uses a harmonic structure and the direction of sound source as cues of segregation. Bi­ HBSS adopts a pair of HBSSes [Nakatani et a/., 1994] for the right and left channel to extract harmonic stream fragments . It determines the fundamental frequency (Fo) of a harmonic stream fragment by coordinating the pair of HBSSes. The direction of sound source is identi­ fied by calculating the interaural time difference (lTD) and interaural intensity difference (liD) of a pair of har­ monic stream fragments of the same F0 extracted by the pair of HBSS. Harmonic stream fragments are grouped by the direction of the sound source.

The residue obtained by subtracting harmonic struc­ tures from an input sound is substituted for non­ harmonic parts of a group. If a group ends with non­ harmonic parts, the residue is substituted for 150 msec. The idea of residue substitution is similar to the psy­ chophysical observation known as auditory induction [Green et a/., 1995; Warren, 1970] . It is a phenomena that human listeners can perceptually restore a missing sound component if it is very brief and masked by ap­ propriate sounds.

3.2 · Automatic Speech Recognition

The automatic speech recognition system, HMM-LR [Kita et a/., 1990], is used to recognize speech streams. HMM-LR is based on hidden Markov model of each pho­ netic transition, in spite of Rodney Brooks' challenge problem. The parameters of HMM-LR are trained by a set of 5,240 words uttered by five speakers.

Since the spectrum of speech streams segregated by the speech stream segregation is distorted due to binau­ ral input, binauralized training data is used to recover from the degradation of the performance of recognition [Okuno eta/., 1996].

3.3 Performance Evaluation

The performance of automatic speech recognition is usually measured by the cumulative accuracy up to the 10th candidate (or simply cumulative accuracy) of word recognition, since ASR returns the first about 10 candi­ dates of each word. Such candidates are further selected by successive speech understanding systems . Therefore, we adopted the same measurement with open tests. By open tests, we mean that the training and benchmark (testing) data are disjoint .

We used three sets of 500 benchmark sounds; one set of 500 two-sound mixtures and two sets of 500 three­ sound mixtures (Table 1). The first sound is uttered by the first speaker at 30° to the left from the center, and the second sound is uttered after 150 msec by the second speaker at 30° to the right from the center (Figure 1). To recognize the first speech in the mixed sound directly by HMM-LR, the utterance of the second speaker is delayed by 150 msec .

The third sound with F0 of 250 Hz is an intermittent harmonic sound from the center. It starts before the first speaker and repeats to last for 1 sec with 50 msec of pause. The average power ratios of the first and second sounds to the third sound in benchmarks Triple and Triple' are 1.7 dB and -1.3 dB, respectively.

The error rate caused by interfering sounds is defined as follows. Let the cumulative accuracy of recognition of original data up to the lOth candidate be CAorg, and let the cumulative accuracy of recognition of (non-binaural) mixed sounds up to the lOth candidate be CAmix · The error rate caused by interfering sounds, £ , is calculated as£= CAorg- CAmix ·

To evaluate the performance of speech stream segregation, error reduction rate is defined. Let the cumula­ tive accuracy of recognition up to the lOth candidate be CA.eg- The error reduction rate, R 309 , is calculated as follows :

The original cumulative accuracies of word recogni­ tion uttered by single speakers, Woman 1, and Woman 2, are 94.99%, and 96.10%, respectively. The error rate by interfering sounds is shown in Table 2.

Error reduction rates by speech stream segregation for the three benchmark sets are shown in Figure 3.3. The Ideal shows the upper limits of error reduction, which are calculated for the case in which the utterances of a single speaker are recognized after speech stream segregation. For Double, 77% of errors caused by an interfering speaker were reduced by speech stream seg­ regation. By additional noise, the SNR of each speech is decreased further (by about 1 dB and 2 dB for Triple and Triple', respectively), but, 55% and 49% of errors are reduced respectively.

Since this performance was attained without using any features specific to human voices, we believe that understanding three simultaneous speeches is a short­ term research problem.

- Double

4 Detailed Plan for the Challenge

The research issues depend on an approach taken by a challenger. Some possible approaches are listed below:

• Either speech separation system or speech stream segregation may be exploited.
• Speech separation or speech stream segregation may run either incrementally or in batch .
• Multiple speech enhancement or separation sys­ tems may run concurrently to extract all speeches or one system may extract all speeches.
• Speech segregation/separation system may be ei­ ther used as a front-end to ASR or integrated with ASR.
• Top-down or hybrid approaches needed for contin­ uous speech recognition or understanding may be employed, although word recognition is requested by the challenge.

We only give a general guideline on the bench­ marks and evaluation criteria in this paper. Fur­ ther information will be made available at the URL of http://www.nue.org/CASA97/ .

4.1 Benchmark Sounds

The common platform for the challenge is quite im­ portant in order to share and transfer the methodology and technology developed by each challenger. Monaural data of speech used for the challenge should should be widely available. The current candidates are as follows:

• The DARPA TIMIT Acoustic-Phonetic Con­ tinuous Speech Corpus for English speeches. It contains a total of 3,600 sentences by 360 speakers uttering the same 10 sentences. (http:/ /www.ldc.upenn.edu/)
• The continuous speech corpus developed by the Acoustic Society of Japanese for Japanese speeches. It contains a total of 9,600 sentences by 64 speak­ ers uttering some of 503 sentences, which were recorded by ATR.

Since these corpus are copyrighted, only the combi­ nation of word utterances will be made available. One benchmark set contains a total of 200 combination of words; three speakers, arbitrary combination of men and women, utter a different word simultaneously.

The acoustic field is made simple enough to produce benchmark sounds easily. A sound source (speaker) should be placed from 1.4 meters to 2 meters from the microphone on the floor in a free-field without rever­ beration. Several combinations of speaker positions se­ lected from 0°, 30°, 45°, 60°, 90°, 120°, 135°, 150°, and 180° may be strongly recommended. The challenge does not assume that any speaker move during speaking . However, challengers may attack the problem of moving speakers.

A mixture of sounds may be either recorded or gener­ ated artificially. The number of microphones should be less than or equal to 3. If a challenger wants to use a binaural input, it may be generated artificially by using Head-Related Transfer Function (HRTF), which speci­ fies the spectral transformation of a binaural sound. The data of HRTF for the KEMAR dummy head microphone is available from MIT [Gardner and Martin, 1994] . For this HRTF, a sound source should be placed 1.4 meters from the dummy head microphone. In our preliminary experiments, all sound sources were placed at the dis­ tance of 2 meters from the dummy head microphone.

4.2 Measurement of Evaluation

The measurement of performance evaluation is error reduction rate as well as cumulative accuracy of up to lOth candidate, both of which are defined in the previ­ ous section. The first measurement may be important because it is rather independent for automatic speech recognition systems used.

The first stage of the challenge investigates the per­ formance of word recognition.

4.3 Tentative Schedule

• The challenge problem will be presented at IJ CAI97 as well as IJCAI-97 workshop on Computational Auditory Scene Analysis.
• The guideline on the benchmarks will be made available by the end of 1997.
• Intermediate progress reports will be presented at AAAI-98 or an appropriate conference in 1998.
• Final progress reports will be submitted in Jan., 1999 and will be presented at IJCAI-99.

5 Conclusions

In this paper, we proposed understanding three simul­ taneous speeches as a new challenge and standard AI problem. It provides rich research issues for a wide range of AI including automatic speech recognition, speech un­ derstanding, and CASA, as well as psychoacoustics. We expect that research on the challenge would play an im­ portant role in realizing the "Prince Shotoku Computer" or powerful computer audition systems.