A Music Stream Segregation System Based on Adaptive Multi-Agents

1 Introduction

In recent years scene analysis based on acoustic infor­ mation, termed auditory scene analysis, has received a renewal of interest. Recognizing external events based on acoustic information is an essential function for sys­ tems that work in the real world.

A principal problem toward auditory scene analysis is stream segregation [Bregman, 1990]. This segregation means decomposing an input signal into signals of in­ dividual sound sources included in the input. However, once multiple acoustic signals are mixed up, their segre­ gation is, so far, considered very difficult because it is an ill-posed inverse problem.

Nevertheless, technical and applicational importance has attracted researchers to this field of study. Specifi­ cally, works intended to model integration of bottom-up and top-down processing includes [Lesser et al., 1993], [Nakatani, et al., 1995], and [Ellis, 1996] . These works are characterized by their architectures based on process­ ing modules with simplified functions and communica­ tions between these modules, which we call a multi-agent architecture. While the architecture intrinsically enjoys modularity and scalability, quantitative background for behavior of agents is not yet established. Practically, the multi-agent based systems mentioned above require sub­ jectively defined rules to control interaction schemes or to adjust parameters for modules in order to deal with variations of sounds.

Here we propose a quantitatively principled architec­ ture, called lpanema, designed to solve the stream segre­ gation problem for sound mixtures. The essential idea is adaptation of agents to cope with variation of a sound. We have developed two kinds of processing in order to realize adaptivity: template filtering and phase tracking. These mechanisms enable each agent to optimally, in the least-squares sense, track the individual sound. As an ex­ ample application of the proposed architecture, a music recognitiqn system has been built. The evaluation tests show that the adaptive mechanisms have significantly improve the recognition accuracy in comparison to a con­ ventional signal-detection/separation method based on the matched filtering.

In the following part of this paper, Section 2 focuses the discussion on the adaptive processing, which is an es­ sential part of the architecture. Section 3 then describes general configuration of the lpanema architecture. After Section 4 introduces evaluation results of the recognition accuracy, Section 5 discusses implications of the present work in the context of existing related approaches. Fi­ nally Section 6 concludes the paper with the expected future work.

2 Adaptation of Templates

2.1 Template Filtering

We consider representing an input acoustic signal z(k) with a sum of template waveforms Yn(k) , where n is the index that corresponds to each sound source and k enumerates sampling time sequences. Our problem can be formulated as minimization of J in the equation:

where N is the estimated number of sound sources, which is not predefined, and E denotes average over time. For Yn(k), we employ one of the simplest models as depicted in Figure 1. Mathematically, the model can be written as

where h is an impulse response of the filter H, r is a template waveform, and M is the length of the impulse response, that is, the number of taps when one use the FIR filter as H.

In this formulation, one cannot predetermine the fixed sets of hand r , because there is a diversity of waveforms even for one specific sound source. For the example of musical instruments, both top two waveforms, (a) and (b), in Figure 2 are piano sounds. Even if we ignore the phase information and consider only spectral power rep­ resentations, the situation is essentially the same because there are a variety of spectrum patterns for one sound source. Therefore we need an adaptive mechanism. Here we would change hn(m). Equation (1) is rewritten using Equation (2) as

The necessary condition for J to hold the minimum value over hn(m) is that the values of partial derivative {)J j{)hn(m) are 0 for all nand m. Using this condition, it is straightforward to derive N x M pieces of simultaneous linear equations as follows,

wherei = {0, 1, ···,N-1} andj = {0, 1,···,M-1}. Since the number of equations (N x M) equals the numher of unknown parameters (hn(m)), the problem is re­ duced to calculation of the inverse matrix, which is a simple algebraic operation.

2.2 Phase Tracking

The condition for the above optimization scheme to. be effective is that the fundamental frequency of each tem­ plate r is exactly the same as the one included in z. This is because a linear filter, H, cannot change the frequency of an input signal. Therefore we need a phase tracking (i.e. instantaneous frequency tracking) method, which changes the phase of template r in accordance with the phase of the corresponding sound source signal included in the input signal z.

If the input signal is not a mixture of multiple sounds but a single sound, adaptive pitch tracking methods al­ ready invented can be used [Nehorai and Porat, 1986]. However, such signal processing methods are not directly applicable to a sound mixture where multiple pitches are present. Thus we have devised a simple algorithm tore­ alize the phase adaptation. The algorithm consists of the following six steps.

1. Perform frequency analysis to the input z, to extract fundamental-frequency components. Because z may be a mixture of multiple sound signals, there may

be multiple fundamental-frequency components.

1. For each fundamental frequency component, choose ri that is a possible template of a sound included in z.
2. Apply a narrow-band bandpass filter tori, using av­ erage fundamental frequency of each Ti as the center frequency of the bandpass filter. For each time sam­ ple, store the phase of the output waveform of the bandpass filter. Let Pr,i(k) denote the phase at time k.
3. Apply the same bandpass filter, as applied to ri, to the input z, and store the phase information for each fundamental frequency as Pz,i(k).
4. Calculate the required time shift .6.kr,i(k). Because the phase difference .6.pr,i ( k) is given as

the time shift .6.kr,i(k) is calculated by

where Is is the sampling frequency and fc,i is the center frequency of the applied bandpass filter.

1. The amplitude value Ti at time k is given as

Figure 3 shows how this algorithm works.

3 lpanema Architecture

This section introduces the configuration of the sys­ tem architecture designed for calculating the scheme de­ scribed in the previous section.

3.1 Overview

A specific feature of the stream segregation problem here is that "signals" or "noises" are not uniquely defined in advance; that is, a stream segregation system is required to handle multiple kinds of "signals" simultaneously 1 .

1 This is a contrast to speech recognition systems where it is defined that the "signal" is a human speech and "noises" are the sounds other than the speech.

Therefore it is a natural idea to build a stream segrega­ tion system by accumulating processing modules, each of which tries to extract a specific target sound as a "sig­ nal" in its charge, regarding other sounds as "noises". In addition, it is obvious that the modules should affect each other in order to create a valid interpretation of scenes.

Thus we propose a system architecture based on the multi-agent scheme, as shown in Figure 4. As an input, the system is fed with a sound signal that is a mixture of signals from multiple sound sources. The current version of the system assumes that the input is an ensemble music signal. As outputs, the system creates a symbolic representation that is similar to the musical scores, and waveforms produced by agents.

The system consists of the following elements: initiator, promoter, !!,gent network, mediator of !!,gents; therefore we call the architecture lpanema. There is also a post-processing module called an information integra­ tor.

3.2 Processing Modules

Initiator

The initiator cuts an input signal into frames and sends the waveform of each frame to succeeding processing. The frame length is variable; when an input signal is available, the initiator tries to find an onset of the sound. Every time it finds the onsets, the initiator creates a new frame.

Promoter

The promoter performs frequency analysis on the wave­ form sent from the initiator and extracts possibly­ multiple fundamental-frequency- components included in the frame. Then it outputs the fundamental frequen­ cies to P-Bus (promotion bus), which is observed by all the agents.

Agents

In our architecture, each agent in the "agent network" is a processing module that corresponds to a single sound source (flute, for example). Each agent has a bank of raw-template waveforms, each of which corresponds to a specific pitch and expression.

Each agent examines the P-Bus information to check the possibility that the sound source corresponding to the agent is included in the input or not. If the agent infers that there is a possibility of being included, the agent suggests waveform ri, applying the adaptive phase tracking method described in Section 2 to one of the raw­ template waveforms. If the agent finds little possibility of presence of the corresponding sound, it just keeps silence.

Waveforms ri are written to the common place called M-Bus (mediation bus) and passed to the mediator. Then the agents wait for the mediator to feed back an­ swers. The answers from the mediator are sets of filter coefficients that optimally modify r;. Each agent reads the answer from the mediator via M-Bus and then cal­ culates an FIR filter with the answered coefficients to obtain a waveform Yi· The final output of the agent is waveform Yi, its average power E[ytJ as activity level, and an information label that the agent is given; for ex­ ample, "Flute C4".

In the current implementation, the agents in the agent network only communicates with the mediator. How­ ever, we expect that the extension of processing scheme would enable communication among the agents them­ selves in the agent network through M-Bus.

Mediator

Here mediation of agents is formulated and reduced to the problem of matrix calculation, as discussed in Section 2. Thus the mediator first receives the r; via M-Bus from agents. Then it calculates the optimal filter coefficients for each agent, using Equation (4). Finally the mediator sends the coefficients back to each agent using M-Bus.

Information Integrator

The information integrator is a post processing module that revises the symbolic output of th€ system. It re­ ceives an information label (e.g. Flute C4) and activity level from each agent, and basically, the label with the highest activity level for each note composes a symbolic version of output of the system. It is inevitable, however, that noises occasionally appear without higher-level in­ formation such as temporal or simultaneous relations be­ tween sounds. Thus the Bayesian networks are employed here in order to integrate multiple sources of information.

In the current system, note transition information has been introduced. The information integrator first con­ structs the Bayesian networks where nodes encode prob­ abilities for the information labels and links represent temporal relation between the nodes. The integrator then updates the probabilities for the labels based on the probability propagation scheme [Pearl, 1986], inte­ grating note transition statistics stored in advance. This paper, however, focuses on the adaptive mechanisms and the details on the information integrator will be reported in a separate article [Kashino and Murase, 1997].

4 Evaluations

We performed two kinds of tests to evaluate the system: a benchmark test for musical note recognition and a sam­ ple song recognition test. In both tests, the information integrator was turned off (i.e. the note transition infor­ mation was not integrated) in order to evaluate basic performances of the adaptive mechanisms.

4.1 Benchmark Test

To evaluate the advantages of adaptive processing de­ scribed in Section 2, we tried to conduct the benchmark test of note recognition used in [Kashino, et al., 1995a; 1995b].

The test signal was a three-simultaneous-notes pat­ tern, as shown in Figure 5. The pattern was composed and created by a computer using digitized acoustic sig­ nals (16bit, 48kHz) of natural musical instruments (flute, piano, and violin). We first recorded the single notes of those instruments at a recording studio and stored the waveforms on a computer. We then mixed the stored waveforms, selecting a designated number of notes. The selection of the notes were programmed to produce the Class-2 note pattern, which is the term of [Kashino, et al., 1995b], where the interval between at least two si­ multaneous notes is a perfect fifth.

We defined the recognition rate, R, as

where right is the number of correctly identified and correctly source-separated notes, wrong is the number of spuriously recognized (surplus) notes and incorrectly identified notes, and total is the number of notes in the input; this is the same definition as in the above mentioned papers. From preliminary experiments, num­ ber of taps of the FIR filter was chosen to be 40 for the template-adaptation-on condition. The template­ adaptation-off means that the number of taps of the FIR filter was fixed to 1.

In this test, if one use the same waveforms as the templates as the ones used for test signals, these wave­ forms will completely match and results will become in-

± : 95 % confidence intervals

appropriate. Therefore we used different manufactur­ ers' instruments, for example, Boesendolfer's piano and Yamaha's piano, in making test signals and templates, respectively.

The results are listed in Table 1, which clearly show advantages of the processing scheme developed in Sec­ tion 2. The condition where both template adaptation and phase tracking are turned off is equivalent to the matched filtering, which is a conventional signal process­ ing method for signal identification.

4.2 Music Recognition Tests

We have evaluated the system using music sound signals. Table 2 lists the note recognition rates for a sample mu­ sic: a live recording of a chamber ensemble "Auld Lang Syne", arranged in three parts and performed by violin, flute, and piano. A part of output of the agents network and a part of score-like data produced by the system are shown in Figures 6 and 7.

5 Related Work and Discussions

For the acoustic signal separation task, much work has been done since as early as 1970's. The approach us­ ing microphone arrays has been one of major research streams [Mitchell et al., 1971] [Bell et al., 1995], and the harmonic selection is another major method [Parsons, 1976] [Nehorai and Porat, 1986]. These approaches have been principally based on a single cue (localization of sources or harmonicity). On the other hand, works try­ ing to integrate multiple cues for stream segregation are

"

Figure 7: A part of results yielded by the implemented system. A note value recognition (e.g. quarter notes, half notes, ... ) is not considered here and all the notes are displayed as quarter notes with a real-time scale.

recently emerging. The most closely related works to the presented architecture are the IPUS project [Lesser et al., 1993], Nakatani et al. [1995], Kashino et al. [1995a] and Ellis [1996] .

The IPUS is an acoustic signal understanding project based on the blackboard architecture[Lesser et al., 1993], seeking adaptive processing according to the input to the system. IPUS realized its adaptivity basically in a rule­ based strategy while the Ipanema architecture does not employ symbolic rules for adaptation.

Nakatani et al. invented a speech segregation system that consists of multiple processing modules called a gen­ erator and tracers [Nakatani, et al., 1995]. The function of the tracer is to trace harmonic structure, which is similar to the function that the promoter in our archi­ tecture performs. However, their system does not have an explicit mechanism to identify sound sources; it only segregates harmonic or localized sound into signals. Our main point is the adaptive processing to trace the specific sound sources, investigating what the sources are.

Ellis proposed a prediction-driven architecture for an auditory scene analysis system [Ellis, 1996], where con­ text sensitivity in scene interpretation is realized. In our current implementation of lpanema, musical context does not affect the template adaptation scheme; the con­ text is utilized in the information integrator.

The studies toward music recognition includes [Mont­ Reynaud, 1985], [Chafe et al., 1985] and [Brown and Cooke, 1994] . However, automatic music transcription systems or music stream segregation systems which can deal with given ensemble music played by multiple musi­ cal instruments with a reasonable accuracy have not yet been realized. A quantitative architecture was proposed by Kashino et al. [1995a], in which a Bayesian proba­ bility scheme was applied. However, their architecture does not yet include adaptive processing and has been applied only to artificial performances synthesized by a sampler [Kashino, et al., 1995a]. The system presented here, on the other hand, was designed for, and tested by, real performances rather than sampler performances.

6 Conclusion

We have pre8ented a new system architecture designed for auditory stream segregation. To cope with a variety of sounds that appear in the real world, two mechanisms, template filtering and phase tracking, have been devised. In implementation, we have taken advantage of modu­ larity and scalability of the multi-agents approach. The adaptivities realized in this paper enable each agent to optimally, in the least-squares sense, track the individual sound included in input sound signals.

As an example application of the proposed architec­ ture, we have built a music recognition system that rec­ ognizes instrument names and pitches of the notes in­ cluded in ensemble music performances. Experimental results show that the adaptive mechanisms significantly improve the recognition accuracy in comparison to the matched-filter-based processing, which is a conventional signal detection/separation method.

This paper has focused on the adaptive mechanisms and left the information integrator, the post-processing module, almost untouched. However, information inte­ gration is an important issue to be addressed: our pre­ liminary tests have shown that integration of the statis­ tical information of note transitions improves the note recognition rate up to approximately 75 % in the same music recognition test as used here [Kashino and Murase, 1997]. To obtain further accuracy, we anticipate that sound source models that explicitly model variations of the sources would be necessary.

Acknowledgments

The authors WOI-!ld like to thank Hiroshi G. Okuno, Takeshi Kawabata; and Tomohiro Nakatani for contributing valuable discussions with the authors. The authors also wish to thank Ken'ichiro Ishii for his support as the Executive Manager of their research laboratory.