Improving Performance of Transfer-Driven Machine Translation with Extra-Linguistic Informatioon from Context, Situation and Environment

1 Motivation

We have been developing a rnethod of Transfer-Driven Machine Translation (TDMT) [Furuse et a!. , 1995][Fu­ ruse and Iida, 1996] in producing an efficient multi­ lingual spoken-language translation system. Systems that deal with spoken dialogs generally require differ­ ent techniques than systems that deal with written lan­ guages. The main requirements for the former are tech­ niques to handle 1) spoken languages containing ungram­ matical expressions, 2) real-time translation to avoid the interruption of smooth communications [Sumita et a!., 1993], and 3) appropriate expressions under environmen­ tally influenced situations.

In TDMT, constituent boundary patterns [Furuse a.nd Iida, 1996] are applied to an input, which contrasts with the linguistic manner of applying grammar rules. The result of doing this provides for robust parsing that can even handle ungrammatical phenomena such as deriva­ tions in metonymical relationships [Iida eta!., 1996]. Ad­ ditionally, by dealing with best-only substructures uti­ lizing empirical knowledge, the explosion of structural ambiguities is significantly constrained. Accordingly, ro­ bust and efficient translation of spoken-language input can be achieved.

In this paper, we primarily review an interim evalua­ tion of the performance of a TDMT prototype system, i.e ., a multi-lingual spoken language translatiOn system in order to establish a criterion for subsequent perfor­ mance improvements .

Additionally, in handling situationally appro~riate ex­ pressions required by spoken-language translatiOn, par­ ticular attention is paid to word usage that depends not only on the linguistic information, e.g., linguistically re­ ferred context but also on the extra-linguistic informa­ tion from the ~ontext, situation and environment in or­ der to achieve natural communications.

In particular , the selection of euphemistic expressions 1 , depends solely on the situational infl.uenc~s. However, although a conventional MT system p.rov1des a large­ scale dictionary, it gives little informatiOn on word us­ age selection for appropriate situations or contexts. The analysis process works only for individual sentences, sep­ arating them from the context.

Thus we describe extra-linguistic information, such as the s~eaker 's role , social rank, and gender, which the system can use to naturalize communications with re­ spect to word usage. We will also proi?<?s~ a meth?d to improve the translation accuracy by ut1hzmg such mfor­ mation.

From the performance improvement of a TDMT sys­ tem that utilizes our method in comparison with the con­ ventional TDMT system, we can state that our proposed scheme can be expected to improve the performance of dialog MT systems.

The next section presents an overview of TDMT and its advantages in comparison with related research ef­ forts. Section 3 describes the performance evaluation of a conventional TDMT prototype system. Section 4 presents our proposed method for improving the sys­ tem's performance together with preliminary experimen­ tal findings and evaluations. In section 6, we state our conclusions.

1 In this paper, we define euphemistic expressions as ex­ pressions paying one's respects to the hearer indirectly, such as by offering superficial choices to the hearer even if the real intention is a command. For example, it is more polite to say "I would appreciate it if you could help me " rather than saying "Will you help me", when you ask something of your

superior, in general.

2 TDMT and Related Research

In attempts to develop a multi-lingual translation system [Lavie et a!. , 1996][Frederking and Brown, 1996][Wahlster et a!., 1993], an interlingua-based ap­ proach has been studied as a theoretically efficient mech­ anism. Several schemes have been proposed recently based on this approach. The semantic pattern-based parsing in JANUS [Levin eta!., 1995][Lavie eta!., 1996] uses frame-based semantics [Goddeau et a!., 1994] with a semantic phrase grammar, and the operation of the parser is viewed as phrase spotting. In this scheme, a recognized speech input is paraphrased actively into a concrete and simple expression that conforms to one of the system's internal representations , which can make the utterance's meaning easy to understand.

Although the above inference schemes are powerful in explaining a speaker's intention or the propositional content of an utterance even from a keyword or phrase, plausible default values have to be prepared for achiev­ ing heuristic inference while extracting the meaning of an input sentence. For example, it is impossible to ac­ cept a sentence that includes a metonymical expression like:

1. "A cheap and clean election is nice."

without preparing features able to bridge the semantic gap between cheap (clean) and election a priori.

Therefore, such an approach may work well within a certain domain, but less scalability may be the result when trying to extend a prototype system in practice. Additionally, even if a semantic concept unit for at least three languages were to be defined, designing a basic generation mechanism able to bridge the semantic gap between languages would still remain one of the more difficult problems.

On the other hand, it is generally accepted that people learn source-to-target expression pairs when they learn a foreign language. Therefore, it would seem practical to design reasonable pairs of expressions for each pair of languages based on a common MT mechanism. In other words, to achieve a practical multi-lingual translation system, it would be reasonable to analyze the source language to suit the target language.

From this point of view, TAG (Tree-Adjoining Grammar)-based MT [Abeille et a!., 1990] and TDMT share many important properties for the successful im­ plementation of practical MT systems. Both schemes produce translations based on the synchronization [Shieber and Schabes, 1990] of the source and target language structures. However, the algorithm for TAG has a worst-case time complexity of O(IGin 6 ), where IGI stands for the size of grammar G and n is the length of the input string. With regard to the speed required for smooth communications, it is unrealistic to apply TAG­ based parsing to practical spoken-language translation systems.

In contrast, TDMT has the advantages of:

• The existence of an efficient parsing algorithm;
• The capability to handle various linguistic phenom­ ena by utilizing translation examples with a simple best-first mechanism.

TDMT's Constituent Boundary Parsing (CBP) can be implemented according to bottom-up left-to-right chart parsing algorithms [Furuse and Iida 3 1996] whose worst­ case time complexity is merely O(n ).

In addition, using best-first syntactic/semantic simi­ larity would make TDMT more flexible [Sumita et a!., 1993].For example, TDMT can accept the previous ex­ ample sentence (1) by utilizing a similar example like "a cheap and clean room would be good" 2 .

Therefore, TDMT can achieve both scalability and efficiency in multi-lingual spoken-language translation. The transfer module, which is an essential component of TDMT, is a common part of the translation mecha­ nism for every language pair, where as language-oriented modules, such as morphological analysis and sentence generation, are provided for each source/target lan­ guage. Furthermore, transfer knowledge is provided for each source/target language pair. TDMT has already been applied to three language pairs: Japanese-English, Japanese-Korean, and Japanese-German.

3 Evaluation of TDMT

The TDMT system, whose domain is travel conversation, is implemented in LISP and runs on UNIX-based ma­ chines. A system dealing with spoken-languages requires a quick and accurate response, rather than a grammat­ ical response, in order to provide smooth communica­ tions. Moreover, since every process, including speech recognition, translation and speech synthesis, runs auto­ matically from start to finish, there is no room for man­ ual pre/post-editing of input/output sentences in order to make the sentence easier for either the translation pro­ cess or the user to read. In other words, assuring both efficiency and acceptability in spoken-language transla­ tion are the most crucial tasks in devising such a system. Therefore, we evaluated TDMT for both speed and ac­ ceptability of translation and analyzed the evaluation results from this point of view.

3.1 The Evaluation Procedure

We evaluated TDMT's translation quality by separately using morphological analysis, a translation module (in­ cluding a generation module) and a parsing scheme (CBP). Manually analyzed morpheme sequences were used to avoid errors and unknown words in testing the translation module itself. This allowed us to assess how well the TDMT would function individually, Details of the evaluation on the morphological analyzer are not de­ scribed here 3 .

Table 1 shows the conditions of our experiment and evaluation. The reader should note that as the JG and KJ translation project have just started, the transfer knowledge will not always be of the same quality for all language pairs. In addition, though the quality of the thesaurus for each language is an important topic for example-based frameworks, according to our experi­ mental results obtained by applying some kinds of the­ sauruses into TDMT, no remarkable differences in the translation quality were observed except for a difference in the number of translation outputs.

2 Although 'room' and 'election' are semantically rather distant, their syntactic similarity enables TDMT to translate the first example sentence into acceptable Japanese while preserving the metonymical nuance of the meaning.

3 The success rate in perceiving morphemes was more than 99%, and that in assigning linguistic categories was more than 98% while the analysis speed was less than 0.2 second for each source language measured using a SPARC Station 10 workstation.

Translations for 69-87 unseen dialogs (about 1,000 un­ seen different sentences) were manually evaluated by as­ signing a grade. Two or three native speakers of each of the target languages performed the assessments; all of the examiners were also familiar with their respective source language in order to judge the correctness of the information.

We used the same dialogs for all of the translations whose source language was Japanese: JE, JK, and JG translations. This allowed us to compare the differences in the training (patterns, examples, etc), the linguistic distance between languages, and so on. Each sentence was assigned one of four grades for translation quality: (A) Perfect - a fluent translation with all information conveyed correctly; (B) Fair- a translation that makes it easy to understand the expressions but with some unim­ portant expressions missing grammatical elements; (C) Acceptable - an acceptable translation; (D) Nonsense, wrong sense - an unacceptable translation or where im­ portant information has been translated incorrectly. The parse structure of each source sentence was also manually evaluated by assigning it a grade indicating: (1) Success or (2) Failure. The translation speed was measured on a SPARC Station 10 workstation.

3.2 Results

Table 2 shows the evaluation results for the TDMT, where "acceptability" is the sum of the (A), (B) and (C) grade sentences. The translation speed does not in­ clude the time needed for morphological analysis. All ratios are taken from the average of two or three exam­ iners. As the table shows, almost 70% acceptability was achieved in the JE result, and almost 60% acceptability was achieved in the EJ result; remarkably, more than 90% acceptability was achieved in the JK translations, although the JK translations needed less transfer knowl­ edge (Table 1) than the others. This observation can be explained from the viewpoint of linguistic similarity; while the Japanese-English (German) language pair is linguistically distant, the Japanese-Korean pair is rather close.

Table 2: Evaluation results.

(b) Evaluation results

The main problem in the translations involved insuf­ ficient examples for CBP. However, an increase in the ratio with the number of examples can be observed in the results. Thus, the total accuracy and acceptability should improve in proportion to an increase in the trans­ fer knowledge 4 . This trend was observed in the parsing quality results. However, in spite of the rate of false pars­ ing, the quality of both the JK and KJ translations were very high. We consider this finding to be due not only to the linguistic distance between the languages but also to the TDMT pattern-based approach having the ability to translate phrases while preserving the nuance even for ungrammatical input sentences.

In addition, some sentences were judged as of the (D) grade because of their lack of adequate politeness in the EJ translation.

Although the speed depends on the amount of knowl­ edge and sentence length, the average times remain less than 0.4 sec; thus, TDMT can be seen as an efficient translation mechanism.

4 Improving Conversational Naturalness

4.1 Extra-Linguistic Information

As we have already mentioned, the selection of eu­ phemistic expressions generally depends solely on situational influences. Similarly, Japanese donatory auxil­ iary verbs such as 'ageru ' have multiple possible expres­ sions depending on the speaker 's honorific attitude to the hearer 5 . Maeda et al. presented a unification based ap­ proach to Japanese honorifics [Maeda et al. , 1988] . Any contextual constraint should be described in a certain domain , that is , in a sub-language, and the real utter­ ance should be expressed based on the appropriate usage under every situation (Fig. 1) . Consequently, a spo­ ken dialog translation system must work in situations where the following are understood: speaker properties (role, gender, and rank), focus in the utterances, dia­ log domains, objects mentioned, actions mentioned, and derivational forms of the words used.

4 This is accepted in general for the example-based frame­ work, since the exact match ratio is certain to increase in proportion to the increase in translation examples. In fact, a total accuracy of more than 93% was achieved in our closed test evaluation for over 1,000 sentences for EJ TDMT trans­ lation.However, we have to ascertain the practical satiation limit , or how much the transfer knowledge can be expanded.

However, whereas a conventional MT system provides a large-scale dictionary, there is little information on word usage under typical situations or context . The analysis process works only on individual sentences by separating them from the context without considering information on the current environment.

In this section , several examples show the typical extra-linguistic information that a dialog translation sys­ tem can utilize to resolve contextual ambiguity with re­ spect to word usage selection , zero-subject resolution, and so on , in order to naturalize conversations. Further­ more, we propose a scheme for utilizing this information in dialog MT systems to enable preferences appropriate for varying types of communications to be established while preserving the efficiency.

4 .2 Socia l Role and Dialog Domain

In the travel dialog domain , a clerk in a hotel, as op­ posed to a traveler, is usually not dining, boarding or using a tourist-oriented transportation system. Such in­ formation can constrain the possibilities of word selec­ tion with respect to polite remarks. For example, the word 'eat ' can be translated into Japanese as 'taberu', 'meshiagarU-honorifi c', or 'itadakU-humble'. However, if the dialog domain is travel, and the speaker is a clerk but not a traveler , 'eat ' is never translated into 'itadakU-humble'. In the same way, we can consider the translation of 'your' in the following sentence:

1. "Could you please tell me your telephone number?" If the speaker's social role is a clerk and the listener 's is a guest , "your telephone number" should be translated as "o-kyaku-sama-no o-denwa-bangou" (guest-polite-possessive telephone-number-polite) .

In addition , it is well-known that the subject tends to be omit ted in spoken Japanese. Thus , having this knowl­ edge would clearly be helpful in predicting the English for missing Japanese subjects. For example, if a cook says ,

1. "M oshi o-meshiagaru no deshitara, sugu tsukuri­ masu"

(if eat-honorific, immediately cook-polite) 6

5 The complexity of word usage selection for verbs of Giv­ ing and Receiving is presented on the web at http: I I cen­ tral.itp.berkeley. edul- ealj lpnotesl donatory_verbs.html.

6 In this paper, sample Japanese sentences are Romanized in italic based on the Hepburn system, and the corresponding

it is appropriate to translate this by supplying "I" as the subject of "eat" and "you" as the subject of "cook".

4.3 Gender

There are several differences between male expressions and female expressions in Japanese. For example, men can use 'boku ' or 'ore' to mean "I" . However, women never use these words in standard Japanese. It has also been reported that women usually use euphemistic ex­ pressions more frequently than do men , and likewise women tend to use softer expressions than do men. Therefore, information on whether the speaker/listener is male or female allows the translation system not only to assign the correct honorific title, such as Mr . or Ms., but also enables the translation to preserve expressions appropriate to the speaker 's gender .

4.4 Socia l Ra nk

There are various types of expressions: polite, honorific, humble, and euphemistic. These expressions are used depending on the social interactions between speakers 7 . For example, in making a hotel reservation,

• ( 4) "How many people will there be?"

'will there be' can be translated into Japanese in at least two forms, 'i-masu-ka' or 'irasshai-masu-ka'. If the lis­ tener in this case is at a socially higher position than the speaker (e.g., the relationship between a guest and a clerk) , the latter expression would be preferred as it bears more esteem. In the same way,

1. "Then , we will confirm your reservation." should be translated into:
2. (5') " Yoyaku-o kakunin-sasete-itadaki-masu" ('reservation-objective', 'confirm-euphemism') by using the euphemistic alternative.

English words with usage modifiers follow in parentheses.

7 Although this type of word usage selection seems a little bit Japanese (or Korean) oriented, the authors thought that handling this kind of difference between languages is very important for smooth dialog translation such as in the case of diplomatic meetings. In fact , some sentences were judged as incorrect translations only because of their lack of ade­ quate politeness, in our experimental evaluation with an EJ translation system.

5 Utilization of Extra-Linguistic Information in TDMT

In this section, we propose a scheme for utilizing extra­ linguistic information in spoken dialog MT systems to enable preferences appropriate for varying types of com­ rr:unications to be established while preserving the effi­ Ciency.

At first, to illustrate the utterance selection mecha­ nism in relation to the extra-linguistic information, we assume that the model is as simple as that shown in Fig. 2. For example, the word 'receive' is translated into the Japanese word 'itadaku' if a more polite expression is re­ quired than the possible translation 'uketoru'. However, the model should independently work for each reason­ able unit within a sentence. This is because each object (e.g. , a person who should be respected) of a predicate of a unit might differ depending on the subject. For instance, let us consider the sentence:

1. "Could you tell me the telephone number where I can contact you please?"

Since the object in the unit phrase "I can contact you" must be the listener , an honorific expression is required in contrast to that of the object in the unit phrase "Could you tell me ... ", which must be the speaker. In order to achieve word usage preferences for each unit phrase of the sentence, we propose a scheme that incrementally applies the model to each phrase transfer of the trans­ lation. In TDMT, constituent boundary patterns are applied to an input sentence, and by dealing with best­ only substructures utilizing translation examples, a de­ terministic translation can be achieved for each pattern incrementally [Furuse and lida, 1996) . Consequently, the design principles of our scheme are:

• Word usage information with relation to extra­ linguistic parameters is attached to each transfer pattern
• The best-only phrase structure selection mechanism using examples and the word usage selection model with extra-linguistic parameters are combined(Fig. 3)
• Candidates for word usage is selected simultane­ ously when each phrase transfer is settled

Furthermore, since this extension preserves the gradu­ ality for phrase chunk transfer in TDMT, it is reason.able to extend our method to simultaneous interpreta­ tion even for lengthly inputs in order to achieve real­ time translation. By utilizing our scheme in TDMT, the previous sample sentence can be translated into more plausible Japanese:

• (6') "o-kyaku-sama-ni go-renraku-dekiru denwa-bangou­
• o uketamaware-masu-ka." ('you-guest-role-objective' 'contact-polite-humble-possible' 'telephone-number-objective' 'tell-me-humble-polite')

5.1 Preliminary Experiment

We conducted a preliminary experiment to examine the performance of this scheme while using the current TDMT EJ system. We first tested principally to improve the politeness with regard to social rank and social role. Each word usage information was installed within the transfer knowledge as follows .

This type of transfer knowledge with a word usage in­ formation for about 50 words was prepared for the exper­ iment. In this preliminary experiment, we assigned the extra-linguistic parameters manually to independently evaluate this scheme. In order to evaluate the effective­ ness ofTDMT while utilizing the word usage preferences, we used 316 of the "clerk's" sentences previously graded (B)(C)(D) from the evaluation results shown in Table 2.

Translation examples of an utterance from the clerk to a guest (modified by") compared to the same utterance from a guest to the clerk (modified by '), are shown as follows.

1. "We have a question for you."
2. (7') Guest-to-Clerk: "shitsumon-ga ari-masu." ('question' 'we-have-polite')
3. (7") Clerk-to-Guest: "o-kyaku-sama-ni go-shitsumon-ga
4. go-zaz-masu." ('you '-guest-role-polite-objectivel' 'question-polite-objective2' 'we-have-humble-polite')

The following results were obtained in the evaluation.

• 115 (36.4%) sentences were changed into other ex­ pressiOns.
• 84 (26.5%) sentences were improved by at least one grade.
• The rest of the changed sentences were still assigned the same grades because of different problems.

By upgrading all results having a (D) grade, the total acceptability can be improved by about 8% using this scheme, indicating that these results are fairly good. However, one problem is the possibility of construct-· ing overly polite expressions in Japanese. This basi­ cally derives from the fact that the current generation of this software does not yet have sufficient knowledge to treat collocational constraints of politeness between con­ stituent boundary patterns. Nevertheless, the problem can be easily overcome by using statistical/stochastic methods such as n-grams or the co-occurrence of phrases. Consequently, it is clear that the proposed scheme should be able to improve the performance of a dialog MT sys­ tem.

6 Conclusions

For handling the situationally appropriate expressions required for spoken-language translation, we have dis­ cussed contextual constraints on word usage that utilize extra-linguistic information. Furthermore, we have pro­ posed a method to improve the accuracy of translations by utilizing this information. Preliminary experimenta­ tion of our new TDMT has shown that the proposed scheme can be expected to improve the performance of dialog MT.

One important area of future research will involve a method to extract the extra-linguistic information from a situation efficiently. Some information types, such as a speaker's social role, should for now be assigned man­ ually in the system. Other information types, such as various preferences for social position, gender and con­ versational domain, can potentially be found in large cor­ pora of linguistic expressions with some kind of stochas­ tic information, such as usage frequency and word us­ age n-grams. Thus, the automatic extraction of extralinguistic information from dialogs by utilizing statisti­ cal/stochastic approaches is one area of interest for fu­ ture work.