A Multi-Agent Computational Linguistic Approach to Speech Recognition

1 Introduction and Motivation

This paper investigates the deployment of multi-agent design techniques in a computational linguistic model of speech recognition, the Time Map model [Carson­ Berndsen, 1998]. The architecture of the model is il­ lustrated in Figure 1. In brief, phonological features are extracted from the speech signal using Hidden Markov Models resulting in a multi-tiered representation of the utterance. A phonotactic automaton (network represen­ tation of the permissible combinations of sounds in a language) and axioms of event logic are used to inter­ pret this multilinear representation, outputting syllable hypotheses [Carson-Berndsen and Walsh, 2000]. In what follows, we firstly introduce the multi-agent paradigm and discuss the use of agents equipped with mental mod­ els. Secondly we illustrate how the model can be recast within a multi-agent architecture. The paper concludes by hightlighting the benefits of such an approach.

2 The Multi-Agent paradigm

The Multi-Agent paradigm is one which promotes the interaction and cooperation of intelligent autonomous agents in order to deal with complex tasks (see [Ferber, 1999] for an overview of the area). The agents used to recast the Time Map model are deliberative reasoning agents which use mental models. The particular architec­ ture chosen to recast the Time Map model is known as the Belief-Desire-Intention (BDI) architecture [Rao and Georgeff, 1991].According to the BDI paradigm an agent is equipped with a set of beliefs about its environment and itself, a set of computational states which it seeks to maintain, essentially desires and a set of computa­ tional states which the agent seeks to achieve, that is intentions.

The agents are created through Agent Factory [ 0 'Hare et al., 1999], a rapid prototyping environment for the construction of agent based systems. Agents created using Agent Factory are equipped with a Mental State model and a set of methods (the actuators). Agents used in speech techonology in the past [Erman et al., 1996] ex­ hibited only weak agenthood. However, the agents deliv­ ered through Agent Factory are intentional agents with rich mental states governing their deductive behaviour; they exhibit strong notions of agenthood. The model presented below is both pioneering and in stark contrast to prior research in that it represents the first attempt to explicitly commission a multi-agent approach in sub­ word processing. The benefits of this model are discussed in section 4.

3 The Time Map Model as a Multi-Agent System

A number of agents are required in order to recast the Time Map model accurately, namely the Feature Extraction Agents, the Windowing Agent , the Segment Agents, and the Chart Agents. These agents are detailed below. The architecture is illustrated in Figure 2.

The Feature Extraction Agents

Numerous Feature Extraction Agents, operating in par­ allel, output autonomous temporally annotated features (i .e. events). These events are extracted from the utter­ ance using Hidden Markov Model techniques, for more details sec [Abu-Amer and Carson-Berndsen, 2003].The feature output is delivered to the Windowing Agent .

The Chart Agent

Chart Agents have a number of roles to play in the syl­ lable recognition process. One role is to inform the Win­ dowing Agent of all phonotactically anticipated phoneme segments for the current window. This information is ex­ tracted from all transit ions traversable from the current position in the phonotactic automaton. For example in Figure 2 an [s] segment has already been recognised at the onset of a syllable. Based on this the Chart Agent can predict a number of subsequent segments, including a [p] as illustrated. Thus, it communicates its segment predictions, along with their constraint, rank, thresh­ old and corpus-based distributional information, to the Windowing Agent.

Chart Agents also monitor progress through the phonotactic automaton by maintaining records of the contiguous transitions that have been traversed as a re­ sult of recognising phonemic segments in the input. The Chart Agent is informed of which segments have been recognised by Segment Agents. If the segment is one which was anticipated (i.e. a phonotactically legal seg­ ment) by the Chart Agent then the associated transi­ tion is traversed.Chart Agents begin tracking the recog­ nition process from the initial state of the phonotactic automaton. If a Chart Agent reaches a final state in the phonotactic automaton then a well-formed syllable is logged. It is also possible that the segment recognised is not one anticipated by the Chart Agent . In this case an ill-formed structure is logged and the Chart Ager1t returns to the initial state of the automaton in anticipa­ tion of a new syllable. In certain cases the Chart Agent receives recognition results from Segment Agents based on underspecified input . In these cases the Chart Agent can augment the results by adding anticipated feature information provided that there is no conflicting feature information in the input . This is known as output ex­ trapolation.

The Windowing Agent

As previously mentioned the Windowing Agent receives segment predictions from Chart Agents for the current window. The Windowing Agent also takes the current output produced by the Feature Extraction Agents, con­ structs a multilinear representation of it, and proceeds to window through this representation. For each win­ dow examined it identifies potential segments that may be present based on partial examination of the feature content of the window. Potential segments can be iden­ tified by using a resource which maps phonemes to their respective features. Priority is placed on attempting to recognise predicted segments first. Therefore, the Win­ dowing Agent spawns a Segment Agent for each potential segment that is also predicted by a Chart Agent , before spawning a Segment Agent for potential segments not predicted by a Chart Agent. The incremental spawn­ ing of Segment Agents for potential segments is depen­ dent on the progress made by Segment Agents already activated. In Figure 2 the Windowing Agent has re­ ceived predictions from the Chart Agent. Having placed an initial window over the feature-extracted multilinear representation of the utterance, the Windowing Agent identifies a number of potential segments in the win­ dow. Segment Agents are spawned, two of which arc illustrated, one for the predicted potential segment [p], and one for the unpredicted potential segment [b]. The Segment Agent is discussed below.

The Segment Agent

Each Segment Agent spawned by the Windowing Agent has a specific phonemic segment which it seeks to recog­ nise. Segment Agents for segments which were not pre­ dicted have default information from the resource which maps phonemes to their respective feat ures. For exam­ ple, according to Figure 2 a Segment Agent attempt­ ing to recognise a [b] requires that voiced (voi+), stop and labial features all overlap in time. However, Seg­ ment Agents for segments which were predicted may have altered rankings, provided by the Chart Agent and founded on corpus-based distributional information or cognitive factors. A Segment Agent attempts to satisfy each of its overlap relation constraints by examining the current window. Each time a constraint is satisfied its rank value is added to a running total known as the Segment Agent's degree of presence. If the degree of presence reaches the threshold then the Segment Agent is satisfied that its segment has been successfully recog­ nised. For predicted segments certain constraints may be relaxed, i.e. not all constraints need to be satisfied in order to reach the threshold. Rather than all Segment Agents reporting back to the Windowing Agent, each Segment Agent communicates its degree of presence to the other Segment Agents in the environment. Only if a Segment Agent identifies that it has the greatest degree of presence and has reached its threshold will it ask the Windowing Agent to proceed to the next window. It will also inform Chart Agents that its segment has been suc­ cessfully recognised. When a Segment Agent finds that one of its constraints is satisfied it can share this result with other Segment Agents. Thus a Segment Agent at­ tempting to satisfy the same constraint may not have to do so.

4 Benefits of the Multi-Agent Approach

The recasting of the Time Map model results in an inno­ vative multi-agent system for speech recognition which is significantly different from more traditional approaches.

This new architecture provides a principled means of dis­ tributing a computationally heavy work load among sev­ eral task-specific agents, operating in parallel, and col­ laboratively, thus alleviating the computational strain. The use of agents facilitates information sharing, search space pruning, constraint relaxation and output extrap­ olation. From a computational linguistic viewpoint this architecture allows an explicit separation of the declara­ tive and procedural aspects of the model. In this way the knowledge sources can be maintained independently of the application which is particularly important in speech recognition applications to ensure scalability to new task domains and migration to other languages.

Acknowledgments

This research is part-funded by Enterprise Ireland under Grant No. IF /2001/021 and part-funded by the Science Foundation Ireland under Grant No. 02/IN1/Il0. The opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of either granting body.