The Preliminary Results of a Mandarin Dictation Machine Based Upon Chinese Natural Language Analysis

Introduction

Today, the input of Chinese characters into computers is still a very difficult and unsolved problem, which is the basic motivation for the development of a Mandarin dictation ~~h~e. We defm~ the scope of ~e ~search by the following limitatiOns. The mput speech Is m the form of isolated syllables instead of continuous speech (The choice of syllables as the dictation unit will be discussed in detail later.). The machine is speaker dependent. The first stage goal of this system is to have 90% correction for the sentences in the Chinese textbooks of the primary schools in Taiwan, Rep. of China. The errors can be found by the user on the screen and corrected from the keyboard. Such a performance is still much more efficient than any of the currently existing input systems. Also, only a small dictionary for demonstration purpose is to be established in the first stage. The machine will work well for sentences formed by the words in the dictionary, otherwise the new additional words have to be keyed into·the dictionary. To our knowledge, this is the first research effort toward a Mandarin dictation machine in the world.

The dictation machine is divided into two subsystems. The first one is to recognize the syllables using speech signal processing techniques, but this is not very helpful at all because m general every syllable can represent many different characters and can possibly form different multi-syllabic words with syllables on its right or left. Therefore the second subsystem is to identify the correct characters from the syllables by forming correct words which is grammatically valid in a sentence and carefully considering the characteristics of the Chinese natural language.

II Consjderatjons for the Specja! Structure of c -bjnese Language

There are at least 80 thousands of commonly used words in Chinese. Therefore the words can not be used as the dictation units. There are at least 20 thousands of commonly used Chinese charaters, each character is mono-syllabic. Each word is composed of from one to several characters. A nice feature is that the total number of different syllables in Mandarin speech is only about 1300. If we use the 1300 syllables as the dictation units, all the words or characters will be covered. However, the small number of syllables implies another difficult problem, that is, many different characters will share the same syllable. This is why we need the second subsystem. Based on the above observations on the special structure of Chinese language, the use of syllables as the dictation unit becomes a very natural choice.

Another very special important feature of Mandarin Chinese languageis the tones for the syllables. Every character is assigned a tone in generf. There are basically four different tones. It has been shown that the primary difference for the four tones is in the pitch contours, and the tones are essentially independent of the other acoustic properties of the syllables. If the differences among the syllables due to lexial tones are disregarded, only 411 syllables are required to represent all the pronunciations for Mandarin Chinese. This means the recognition of the syllables can be divided into two parallel procedures, the recognition of the tones, and of the 411 syllables disregarding the tones.

Ill The Oyeral! System Strycture

Based on the considerations described above, the overall system structure for the Mandarin dictation machine is shown in Fig.l. The system is basically divided into two subsystems. The first is to recognize the syllables, and the second is to transform the series of syllables into the characters. For the frrst subsystem of syllable recognition, the corresponding syllable (disregarding the tones) and the tone are then. recognized independently in parallel. Because errors always happen, we therefore have to provide information for confusing syllables, and confusing tones. For the second subsystem we need to first form multi-syllabic words from each syllables. To use the above example, although there are many characters all correspond to the syllable [guo-2] and many to [iu-3], there is only one multi-syllabic word " 1!1~8 (Mandarin)" has the pronunciation" [quo-2] [iu-3] ",etc .. But this doesn't solve the problem well. First, mono-syllabic words, such as [ni-3], [shr-4], [i-2], [jia-4], [huei-4], [tieng-1] can't be identified in the above way. They even form ambiguous multi-syllabic words, for example, the syllable [i-2]

can combine with the syllable to its left Lshr-4] to form a wrong word" Ji!i1i: (suitable, [shr-4] [i-2])", or with the syllable t? its right [jia-4] to form a wrong word " l~~ (Move, [I-2] [jia-4])". The problem becomes even w?rse when errors ?Cc~r in the recognized syllable, for example ·. If t~e syllable [m-~] IS incorrectly recognized as the ~yll~ble [m-4], It can be combme~ with the syllable [shr-4] to Its nght to form a wrong ~ord J2!~ (Bad Situation, [ni-4] [shr-4])". The next operation of syntactic analysis then serves as a filter .to rule .out all ungrammatical combinations ~f mulu-syl.labic a~d mono-syllabic words and only a smgle syntactically valid sentence will be obtained and appear on the screen as the output text.

IY The Recognition of the Syllables

The recognition of the 411 syllables in the first subsystem is in fact very difficult2,3. This is because the 411 syllables consist of about 37 confusing sets 3 each of which has from about 4 to 19 confusing syllables2· . A good example is the A-set: ([a]. [ba] , [pa], [rna], [fa]. [da], [ta], [na]. [Ia]. [ga], [ka], [ha]. [ja], [cha]. [sha] [tza]. [tsa], [sa]} . If we use 200 of these syllables to train a commercially available speech recognizer product with 200 words of recognition vocabulary designed for English words, the recognition rate is on the order of 54%- 60% only, although the rate is about 92% for English words. An initial I final two-phase recognition approach is thus specially designed2,3 to recognize these very confusing syllables, whose block diagram is in Fig.2. We first detect the final part and recognize the final from a total of 38 different finals. Once the final is determined, we then try to recognize the initial preceding the final among at most 19 candidate initials.

The recognition of the finals is also very difficult because there are again many confusing final sets, such as ([a]. [ia]. [ua]}. etc. In our research it was found that using multi-section vector quantization (MSVQ) techniques with branch-and-bound algorithm operated on the filter bank coefficients can give very good results while maintain relatively simple hardware/software implementation 4 . Preliminary result of recognition rate of 93.4 % has been achieved for speaker-dependent case. The recognition of the initials is even more difficult because they are relatively short, unstable, and confusing. A special two-phase recognition approach is therefore developed to recognize the initials 2 ·3. The first phase classifies the possible unknown initials into several small groups using formant parameters. The second phase then recognizes the exact initial. The preliminary recognition rate is found to achieve 90.6% for speaker dependent case. The correct recognition of the tone is also difficult. The primary problem is the confusion caused by the third tone. We use here the tone recognition method developed by J.-C. Lee 5 , in which the sum and difference of the log pitch frequencies for adjacent frames are taken as feature parameters. Vector quantization (VQ) codebooks and Hidden Macov Models (HMM) are developed to recognize the tones. The recognition rate for the tones is found to be 98.3%. Combining the results for the final, initial and tone recognition, the total recognition rate for the syllables is only on the order of 82.3%.

y Word Formatjon from Syllables

This is the first part of the second subsystem. The output characters and words for the above example, [ni-3] [shr-4] [i-2] [jia-4] [huei-4] [tieng-1] [guo-2] [iu-3] [de-5] [dian-4] [nan-3], is shown in Fig.3. First, all multi-syllabic words which can be found in the dictionary will be obtained, such as " li!i1i: (Suitable, [shr-4] [i-2])", " lH~ (Move, [i-2] [jia-4])", "~~g Mandarin, [guo-2] [iu-3])", " 11~ (Computer, [dian-4] [nau-3])". Secondly, all syllables which can not form multi-syllabic words with adjacent syllables must correspond to mono-syllabic words, therefore the most frequently used and second frequently used (if any) mono-syllabic words will be found, such as " iw- (You)" and "ltf (Plan)" for [ni-3], " fl (Can)" for [huei-4]. etc. Thirdly, if some of the syllables in the multi-syllabic words obtained in the first step correspond to very frequently used mono-syllabic words, such as " ~ (Are)" and " $ (Affair)" for [Shr-4], etc., they are also given. In the dictionary, the words are organized in a special structure called word trees such that all words having the same first syllable will be organized in the same tree, and all words in the same tree having the same second syllable will be organized under the same node, etc. to improve the searching speed. In order to reduce the total number of words which have to be stored in the dictionary, some additional word formation rules are designed to generate new words6. For example, a verb followed by the character " It!} " ( [guo-4] ) simply represent the past tense, such as " OZ. " ( [ chr-1], eat ) and" It!} " forms " OZ. It!} " ( [chr-1] [guo-4], ate), etc. Some additional word deletion rules are also designed such that redundant or impossible words can be automatically deleted without going to the syntactic leveJ6. One example rule is as follows. Any shorter words within a longer word should be deleted. For example, the shorter words " 1:i'r.l! " ( [tai-2] [wan-1]. Taiwan) and" :k~ " ( [da-4] [shiue-2], University ) will both be deleted if the longer word " 1:i'i\!! :k~ " ( [tai-2] [wan-1] [da-4] [shiue-2], Taiwan University) exists. Another example rule tells that all words which can not be connected by other words on both sides (except beginning and ending words) should be deleted.

VI Syntactic Analysis To Jdentjfy Correct Words and Characters

This is the second part of the second subsystem. The input is the series of all possible mono-syllabic and multi-ayllabic words corresponding to the recognized syllables. The goal is to find the unique correct combination of these words which form a grammatically valid sentence. The approach is simply to take all possible combinations of the words and parse the sentences in parallel based on a Chinese grammar. The ungrammatical combinations will be ruled out and only the correct solution will be left and shown on the screen. A Syntactic Analysis System for Chinese Sentences (SASC) is employed here to parse theJJOSsible sentences and rule out ungrammatical sentences 7 ,If, In SASC, Chinese sentences are syntactically analyzed from the viewpoints of generative grammar. It uses a bottom-up parser instead of a top-down parser, because the former tends to be more efficient for Chinese sentence analysis. The parser uses charts as global working structures to avoid the inefficiency in duplicating many computations that a top-down parser often suffers when backtracking occurs. The parser parses sentences in such a way that phrases are built up on the chart by starting with their heads and adjoining constituents on the left or right of the heads. For example, according to the phrase structure rule (PSR), "VP -----> V-n NP", V-n (transitive verb) is the head of VP(Verb Phrase). When encountering a transitive verb, the parser's action is to try to adjoint the following NP as its object. In order to solve the complicated syntactic phenomena such as passivization, relativization, topicalizati.on, ~nd Ba-transformation in Chinese sentences, a ra1se-bmd mechanis111 based upon the theory of empty categories is developedK. In this way, the SASC parser will treat the above different transformations very easily in the same way, i.e., if an empty element is inserted into the right position, the syntactic structure will become easy. The parser generates an empty NP inserted into the vacant position where an NP is expected to appear. Using this approach, many syntactically sophisticated sentences can be easily parsed.

yu Test Results and Conclusjon

By combining all the different operations described above to form the complete system, some preliminary tests are performed. The results in general depend on the text. The rate for syllable recognition is on the order of 81% - 83%, but the final rate for dictation (i.e., the correction rate for the characters) is on the order of 88% -90%. In other words, some errors made in acoustic level can be finally corrected in the syntactic level. Further improvements should be made on each part of the system and the project is still under active development.

Acknowledgement

The authors wish to thank many excellent students working with them in Taiwan University in the last few years. Without their diligent efforts such research results are simply impossible.