An Understanding System of Natural Language and Pictorial Pattern in the World of Weather Report

1. Introduction

The present paper is based on the experimental res earch for understanding system of natural language and pictorial pattern. The prototype system is called ISOBAR(Information understand­ ing System Of BAsic weather Report). The model was designed and developed after the following phylosophy:

1. In order to be able to understand natural language for human and machine, they need to have some correspondence to the knowledge or experience of external world(in our case visual world),
2. And also in order to make them understand pi c torial patterns, it is necessary to extract and recognize s ome charac·teristic features of pa ttern, corresponding to human concepts which r ef lect to the struc ture and meaning of natural l anguage man spont aneously developed in his long history,
3. Through the s e i nteractive mechanism, man can explain the pi ctorial pattern by using na tural language and vi ce versa.

The ISOBAR's wor l d is limited to only Japanese sent ences of weather report and weather chart of J apan and Far East Asian areas. But from the view point o f s emantics, the fundamentals of processing algor i thm or mechanism of this limit ed world is applicable to more general wo rld.

ISOBAR has two ope rating modes i n princ iple .

Thi s work was financ ially support ed in part by Grant-in-Aid for Special Proj ec t Resea r ch of the Ministry of Education, Sc i ence and Culture, ·1978. Projec t Numbe r 310233.

One is accumulation of meteorological database both from linguistic and pictorial inputs. The other is its retrieval. In the accumulation mode, man presents weather report sentences and charts to the machine. The machine interprets and understands the input information semantic­ ally by the aid of general knowledge about meteorology. It transforms the input into suitable data format and adds to the database if and only if the input is meaningful and conveying new information. If not, the machine either inquires about ambiguous points, or rejects the total input.

The semantics in ISOBAR in the final version is des igned to have such hierarchy as in Fig.l. GKS is the symbol for General Knowledge about Semantics, and GKM for General Knowlcdg£> about Meteorology. SWM and KMP arc S£>mant ics in till' World of Meteorology and Knowledge about Mctl·o­ r ological Phenomena respectively. Generally speaking, it is difficult to distinguish GKM from GKS strictly.

The present ISOBAR is not completed as a total s ys tem with full semantic hierarchy as in Fig.l. Sl.JM was already implemented in the present system because it is the core of the semanti cs in the system, and fortunately it has rather simp l e structure. In the meantime, GKS must cover all field of our linguistic activities in Japanese. So in our case, GKS is limited only to general linguistic knowledge which is essen­ tial to the field of weather report. Present ISOBAR is partly containing GKS in the form of "Event of Constituent" which will be introduced in the next chapter. KMP is regarded as the total contents of an "Encyclopedia of Meteorol­ ogy", but this part is limited and is not expl­ icitly separated from SWM in the present system.

2. Outline of ISOBAR

First, the weather report sentences and weather charts for ISOBAR are outlined. Then some important semantic notions are presented here.

2.1 Sentences of weather report

We limited the words and sentence patterns for ISOBAR only to those of weather reports in Japan which are broadcasted every day from NHK on its second radio program. We termed them basic weather reports. NHK weather reports consist of four parts:

Part 1 General weather conditions

Part 2 Local weather conditions

Part 3 Reports from vessels

Part 4 Information for fishermen

Among these, we omitted the sentences in Part 1 from our consideration. All of these materials were collected from real NHK program between June, 1976 and May, 1977, and these were stored in the database of ISOBAR. Examples of sentenc­ es from Part 2 to Part 4 are shown in Fig.2.

2.2 Weather charts

Weather charts for ISOBAR are illustrated in Fig.] . The me~ning o f each pictorial component will be found in Table 1.

2.3 Notions in the semantic structure of weather report[l],[2],[1)

We introduce three basic semantic notions to

Part 2)

O-SAKA DEWA NNE NO KAZE , FU-RYOKU 3 TEN-Kl WA HARE DE , KIATSU 998MB , KION- 23"C . (OOSAKA area, NNE wind, wind force 3, fair, pressure 998MB and temperature 23"C . )

Part 3)

KAN-TO-TO-HO- NO N35"El46" DEWA W NO KAZE , FU-RYOKU 4 , TEN-Kl WA HARE DE , KIATSU 05MB . (At N35"El46" east of the KANTO district, west wind, wind force 4, fair and pressure 1005MB. )

Part 4)

• (a) KAN-TO-HICASHIKAIJO- NO N36"El49" NIWA , 1004MB NO TEIKIATSU CA ATTE , E Nl 75KH DE SUSUN-DEIMASU (A low pressure of lOOfi MB nt N36"El49" east of the KANTO district, proceeds east at 75KH/hr.)
• (b) NIHON-HUKIN- 0 TO-RU 1020MB NO TO-ATSUSEN- WA , N60"El40° , N56"El36" , · · · , Nl9"Ell3" NO KAKUTEN0 TO-TTEIMASU . (An isobar of 1020MB in and around Japan passes through N60"El40", N56"El36", ··· and Nl9"Ell3" .)

Fig.2 Examples of weathe r report sentences describe the semantic cont ents of the world of weather report. The first is "Constituent " of meteorological world in ISOBAR. The second, "Attribute" of Constituent, and the las t, "Event" in Attribute-space. "Attribut e-space" in this cont ext means an abstract space whose bases are all Attributes essent ial to that Event .

After careful analysis of weathe r r eport senten­ ces and weathe r charts, we postul a ted i7 Consti­ tuents(denoted by C1,C2 ,··· ,C1 7) in a l l. These are tabulated in Table 2. Attribut es ot Consti­ tuents are extracted to be 14 di ffe r en t kinds (A1 ,A2 , ... ,A14 ) as in Tabl e 3. Each Attribut e is, in other words, a semantic phase or point of view of Constituents. Relation be tween Constituents and Attributes makes "Cross­ Relation Table" of Constituents and Attributes. "Semantic Table" is defined t o be such Cross­ Relation Table that is filled wi th each "Att rib­ ute Value". A set of weather repor t s at a special data and time, or a sheet of weather chart, makes complete Semanti c Tables, if all input sentences or pictorial component s are meaningful, and all are recognized correc t ly. Table 4 shows an example of Semanti c Tables.

"Event" in an Attribute-space is introduced as a semantic contents of "verb", "adjective", " ad­ verb" , etc.; thos e are all representing the conditions of Constituents or relation among Constituents. We visualize each event as a locus of some Constituent in an Attribute-space. There are many patterns of Events in the meteo­ rological world in ISOBAR. Some are temporal patterns, others are spatial patterns. In the The tense of S2 is "Progressive present tense", which we denote by the pair of times, i.e. t 0(the present) and t 1(some . immediate future).

linguistic analysis of weather report, we need an explicit expression of each event pattern. ISOBAR has all lists of variations of event patterns, which appear in the world of weather report. These event patterns work as both GKS and SWM in the present system. Semantic Tables, in the meantime, exclusively work as SWM.

2.4 General performance of the system

The total configuration of ISOBAR is illustrat­ ed in Fig.4. It consists of seven processing units and three databases. Linguistic input, i.e., weather report sentences in the accumula­ tion mode, and questions in the retrieval mode, are sent to STA(Syntactic Analyzer) unit first. If the input is syntactically correct, it is stored in the LDB(Linguistic DataBase) only in case of weather report sentences, and at the same time, it is transferred to SSG(Semantic Structure Generator) unit. SSG generates the semantic structure of the input sentence. Then SAS(Semantic Analyzer and Synthesizer) analyzes the structure. The output from SAS makes a Semantic Table which is to be stored in SDB(Se­ mantic DataBase) in the accumulation mode, or to be hold in SAS itself in the retrieval mode.

On the other hand, pictorial input, i.e., the weather chart, both in the accumulation and retrieval mode, are given to PPP(Pictorial Pre­ Processor) first. Pictorial processing consis­ ts of digitization of analog image, noise elimination, data compression, etc .· In the accumulation mode, all original data and their second data are stored in PDB(Pictorial Data­ Base) in a compressed form according to a special coding scheme. Output from PPP is sent to PR(Picture Recognizer) unit in every mode, where each component in the chart is recognized. Then its output is fed to SAS unit. SG(Senten­ ce Generator) and CG(Chart Generator) work only in the retrieval mode. That is, linguistic answer to the question is synthesized by SAS using SDB and generated by SG, while pictorial answer is synthesized by SAS using SDB and PDB, then generated by CG. More details about these units and databases will be presented in Chapter 3 and 4.

Fig.S shows the hardware system of ISOBAR. The main system has special I/O devices, all of which are peripherals of either T-40(TOSBAC-40C) mini-computer or 0-4300(0KITAC-4300b) multi-CPU mini-computer system.

3. Sentence understanding

The outline of the processings in ISOBAR for linguistic input is presented in this Chapter.

3.1 Restriction for input sentences

Original weather report sentences have mostly regular sentence patterns. In ISOBAR, we admit any sentence which satisfies the following restrictions.

[Restriction 1] The vocaburary for input senten­ ce is about 200 words, most of which are ~dopted from original weather report sentences, and some specific words are supplemented for interrogative and imperative sentences, e.g., "ITSU(when)", "DOKO(where)", etc ·.

[Restriction 2] The forms of input sentences are either (!)imperative form like, "···GATA NO TEN-KIZU 0 SYUTSURYOKUSEYO(Output all weather chart of ··· type.)", or (2)such forms that are arbitrarily transformed from original kernel sentences in weather report by the optional rules Rl-RS.

Rl:Alternation of words in kernel sentences. R2:Addition to such a word-group as "1976 NEN- 9 GATSU 11 NICHI 12 JI (WA)", which denotes a special data and time.

R3:Substitution of interrogatives to words or word groups of kernel or transformed sentences.

R4:Abbreviation of words.

RS:Addition of interrogative particle "KA". [Restriction 3) Input Japanese sentences are segmented word by word with spaces.

3.2 Syntactic analysis, semantic s tructure generation and semantic analysis

The functions of STA and SSG in ISOBAR are as follows.

Input sentences are transformed in STA from surface structures into -dependency structures by steps (1)-(4) below, and then into semantic structures in SSG by step (5).

Step(l) Parsing into individual words.

Step(2) Forming word groups.

Step()) Construction of inter-word dependen­ cies within each word-groups.

Step(4) Construction of dependencies among word-groups.

Step(S) Transforming dependencies i nt o sema­ ntic structure(i.e., Semantic Table) by referring the Dic tionary of Conceptual Connection(DCC).

At the end of step(4), each input sentence is identified with one of the template patterns, where verbs are essential keys. DCC as above is the most important dictionary that indicates which word-groups are representing "Constituen­ ts" or "Attribute Values", and in which slots in the Table they are to be stored.

In the course of above steps, syntactically anomalous sentences are rejected in STA. Sever­ al types of anomaly are detected in SSG, and in this case no semantic structures are generated. The semantic structures just generated ~re ana­ lyzed by SAS, and semantic contradictions are detected which can not be interpreted or under­ stood in the world of weather report. This is realized as follows. SAS examines eve ry slot of the Semantic Table whether stored Constitue­ nts and Attribute Values are correct. If not, the semantic structure is contradictory and rejected.

3.3 Access to SDB and output generation

The response of the system to a legal input sentence is determined according t o kinds of input sentences . It is as follows. Case(l) is the only case that opens the accumulation mode of ISOBAR, while Case(2)-(4) open the retrieval mode.

Case{l) Declarative sentence

Every declarative sentence is assumed to be asserting some facts about weather. SAS compares the generated semantic structure of the input with every structure that was already stored in SDB, where comparison is executed on every Attribute. If the input is conveying new infor­ mation to SDB, it is stored there, otherwise SG comments that it is already stored or contradictory.

Case(2) YES-NO question

Case{3) WH question

Case(4) Imperative sentences

If an imperative input sentence demands all charts of a specific type of pressure configur­ ation, the system interprets the input as "When did the pressure configuration of···type occur?". Examples are shown in Appendix A.

4. Picture processing

In this chapter we describe the operation of PPP and PR in ISOBAR.

4.1 Pictorial pre-processor(PPP)

An analog image is converted into binary-valued picture with 1,024xl,024 pixels by PDS-camera. The noises caused by quantizing operation, are eliminated by simple masks. The pre-processed pictorial data are sent to PR. At the same time, they are stored in PDB. In this case, we took a strategy to reduce the amount of storage. In our working system, a picture is represented by a series of black or white square regions of various sizes. The square size, its location, and the gray level(either black or white) are encoded in the structural information of "0", "1" and "(" series. We call this expression DF-expression[4],[5]. OF-expre­ ssions of original charts, together with their second data, are stored in PDB. The second data include such items as complexity of picture, spectrum of primitives, headings of weather patterns, etc .· These are used for chart sear­ ching operation in the retrieval mode.

4.2 Picture recognition(PR)

The major function of this unit is to extract and recognize each picture component as shown in Table 1. We have some a priori knowledges about weather chart in PR. They are as follows.

• {1) Every weather symbol is enclosed with a circle of a constant size. The variation of symbol is limited only to its interior pattern.
• {2) Other components are not connected to the circle except wind symbol.
• There are 15 weather symbols in a chart. The location of respective symbol is fixed.
• The location of coast lines, latitudes and longitudes are fixed, but they are partly erased or interrupted by other components.
• Only front symbols and a few kinds of weather symbols have areas. The others are line drawings.
• Numerals are located near the characters or isobars. They specify the number of Typhoon or pressure values.
• The shape and size of numerals and characters are normalized.

The flow diagram of PR is shown in Fig.6, where each number within parentheses corresponds to above a priori knowledge which is used there. The final output from PR makes the Semantic Tables. Thus, all information from a weather chart is recognized as the semantic content of the world of weather report at the given data and time. We illustrate the results of step(l) -step(4) in Appendix B. These were experiment­ ed on a part of an entire picture covering Western Japan. The picture size is 256x256.

5. Discussion

According to our experience, the performance of ISOBAR in the accumulation mode for linguistic input was very satisfactory. Also in the retrieval mode for linguistic input, the system worked successfully. In other words, we can say present restrictions for input sentences are rather severe. So do the vocaburary and senten­ ce patterns. We should remove such restricti­ ons as much as possible in the next version of

ISOBAR.

On the other hand, its performance for pictori­ al input was not so much successful. There are no fundamental problens left for the performance of pictorial output, that is, the retrieval operation. But in the accumulation mode, or in recogn i zing operation of the system, some prob­ lems are left unsolved. For instance, computa­ tion time is the most untolerable problem. This is partly because of our small(in terms of the size of the main memory) and slow(cycle time) computer system, and partly because of large size of input pictures in spite of rather high compression. Another problems left are improv­ ement of curve following algorithm. Present ISOBAR work~ in the man-machine interactive mannar, but we must find some good idea of auto­ mati c gap tracer. Any solution, if ever, of those and other unsolved problems seems to require a high performance machine.

6. Conclusions

The conc lusions obtained from this research are as follows.

1. The semantic model in ISOBAR was proved to be appropriate.
2. The system performance for linguistic input was satisfactory.
3. The present ability of ISOBAR for pictori­ al input is not necessarily sufficient, because of the machine size and some unsolved problems.
4. In order to make ISOBAR more flexible and powerful, it is desired to remove restrictions for input sentences as much as possible.
5. All pic ture processing procedure should be improved to be more time saving ones. But now, authors have already started such improvement.

ACKNOWLEDGEMENT

The authors are deeply indebted to Mr. Y.Ogawa, R.Taniguchi, K.Tsuzuki and K.Noda for their considerable assistance with the experimental work.

APPENDIX A Examples of Q-A

1978/12/30/12/ WA NIHON-KAIGAWA DEWA TEN-KI HA DAITAI HARE DESU KA. (Was the weather on the side of the Japan Sea almost fair at noon on Dec. 30, 1978?) SPATTERN=7 QUEST=l PLACE=l2 WEATHER=2 DEGREE=2 HAl , NIHON-KAIGAWA DEWA TEN-KI WA DAITAI HARE DESU. {Yes, the weather on the side of the Japan Sea was almost fair.)

1977/1/12/12/ WA NAHA DEWA TEN-KI WA S KA. (In NAHA area at noon on Dec. 12, 1977, was the weather south?) ZOKUSEI(TEN-KI) TO ZOKUSEICHI(S) TOGA HUTEKIGO- . (The weather has not the attribute of direction.)

1976/9/12/12/ WA L NO CHU-SHIN- KARA ON-DAN-ZEN-SEN- GA OOKO Nl NOBITEIIIASU KA. (Where did a warm front extend from the center of the low pressure at noon on Sep. 12, 19767) SPATTERN·4 QUEST· l TJME·1976/9/12/12/ ORG=L FNAME·S LOC=# L NO CHU-SHIN- KARA ON-DAN-ZEN-SEN- GA N26DE140D Nl NOBITEII-IASU . {A warm front extended to N26°El40° from the center of the low pressure . )

ITSU SEIKO-TO-TEI NO KIATSUHAICHI TO NATTEIMASU KA . (When

did the pressure configuration of typi cal winter type occur?) SPATTERtl=6 QUEST=l TIME·# TYPE=l 1976/11/29/18/ \lA NIHON- HUKIN- WA SEIKO-TO-TEI NO KIATSUHAICHI TO NATTEIMASU . (At 6:00p .m. on Nov . 29, 1976 , the pressure configuration of typical winter type occured a round Japan. )

APPENDIX B Experimenta l r esul t of PR