A Parsing Method for Identifying Words in Mandarin Chinese Sentences

Abstract

This paper presents a parsing method for identifying words in mandarin Chinese sentences. The identification system is composed of a Tomita's parser augmented with tests originally a part of the EnglishChinese machine translation system CCL-ECMT together with the associated augmented context-free grammar for word composition. The simple augmented grammar with the score function effectively captures the intuitive idea oflongest possible composition of Chinese words in sentences and, at the same time, take into consideration the frequency counts of words. The identification rate of this system for the corpora taken from books and a newspaper is 99 .6%. This identification system is simple, but the identification rate is relatively high. The minimum element for word-composition parsing is down to characters as opposed to sentence parsing down to Chinese words. It has the potential of incorporating phrase structures and semantic checking into the system. In this way, word identification, syntactic and even semantic analysis can be organized into a single phase. The results of testing the word identification on corpora taken from books and a Chinese newspaper are also presented.