A Plot Understanding System on Reference to Both Image and Language
Proceedings of the Seventh International Joint Conference on Artificial Intelligence. Natural Language. Pages 77-83.
© 1981 International Joint Conferences on Artificial Intelligence
Abstract
In tl\is paper, a. system is described tl\a.t ca.n understand a. plot of a. story on reference to botl\ ima.te and linguistic information. As input, a. series of line drawings witl\ colors and narrations in Entlisl\ concerning to tl\ese drawings a.re given to tl\e system. It searches tl\e objects suggested to be in tl\e scene by tl\e narrations, finding relations among tl\em, ma.kinll tl\e world model by usinll its world knowledte. Its refernce to tl\ose drawings makes it easy for system to ana.lize complicated structures in tl\e narration sentences sucl\ a.s tl\ose of prepositions, and tuides tl\e process rea.soninll about tl\e CD representation usinll rules and demons. At the end of tl\is paper, a result on QA will be sl\own
- * R)~RESENTATIONS ** ll 108,\Q =P= PTRANS -o-OBAQ -D-OlPL liiHEREll 2l IOBAQ =P= Pfi'RANS -0-API'LF.51 -R-OUilll OBAQll 3l (($POSl APPLE! ON TREE! +.723:J29E+08 +.264331E••00l 4l (I *I'OSl APPLI!:2 ON TIU:E I + . 21U1823E+00 + . 23B853E+00l :Jl <<*POSl APPLE3 ON l~EI +.43:J294E+00 +.:J63694E+OOl 61 «*POSl APPLE51 ON TREEll 7) ( 084Q = P= A TRANS -0-STONE I -D-Olllllll OBAQ) = R= < 12l l 121 <OBAQ =P= SMOVE -0-STONE! -D-(lllllllll ::<I'Lll =E= <71 •RI= 16 =R= ( 18 23 26 3lll 13) <<*POSl APPLE! ON TIU:El +.71~169E+88 +.271~42E+OOl 141 ( < *I'OSl APPLE2 ON TREt:t + . 26~62:JE+88 +. 245{)96E+80l l~l ((*POSl APPLES! ON TREEll 161 <OBAQ =P= PROPEL -0-APPLE3 -D-((+.177382E+03 +.137341E+83 +.977839E+80l <+ . 196673E+03 +. 173271E+03 +.62060:JE+08ll -1-121 171 <<*POSl APPJ.E3 CONTACT OBAQ + . I09574E+OI +.7437:JOE+08l 161 <OBAQ =P= ATHANS -0-APPLE3 -D-(lllllllll OHAQl =E= ( 12) =R= 123 26 31ll 19l <<*POSl APPLE! ON TllEEI +.743055E+00 + . 265100E+00l 201 <<*POSl APPLE2 ON TREE! +.312:J00E+00 +.244966E+00l 21 l ( < *POSl APPLE:J I ON TR):Et l 221 <<*PSSl HOUSEl OHAQl 231 IOBAQ =P= Pl~S -0-APPLE3 -D-(lllllllll HOUSEl) -1-24 =E• ( 12 16ll 241 <OBAQ =P= PTRANS -0-OBAQ -D-(lllllllll HOUSEill 2:Jl <OBAQ =P= PTRANS -0-OBAQ -D-llllllllll I INSIDE HOUSEilll 26l IOBAQ =P= PTRANS -0-APPLE3 -D-llllllllll liiPLl =E= < 12 16ll 271 <<*POSl APPLE3 ON TABLEt +.456522E+00 -.215909E+00l 261 <OJIRO =P• PTRANS -0-OJIRO -D-(lllllllll OBAQl -1-29l 29l <OBAQ =P= PROPEL -0 -VOICE-D-<OBAQ OJIROll 301 I<*POSl APPLE3 ON TABLEt) 31l <OBAQ =P= ATRANS -0-APPLE3 -R-<OBAQ OJIROl =E= ( 12 16ll 321 <OJJRO =P= INGEST -0-APPLE3 -D-<MOUTH STO~CRll References (I )R.Schank : Conceptual lnf ormation Processing, North- Holland Publishing Co .Ltd. 1975
- W.lehnert : H•1man and Comp!'latlonal Question Answeri ng, Congnltive Scler.ce I, 47-74(1977)
- E.Charniak : Toward a Model of Children's St o r y Comprehension, AITR-2666, MIT, ( 1972)
- J .Carbonell : Toward s a Process Model of Hum an Personality Traits, Artificial Intelligence 15 , 49-74 ( 1980)
- R.Schank : Controling Inference, Artlflcall Intelligence 12 , 273-297 ( 1979)
- R .WIIensky : Why Jhon Married Mal'y : Under r tand i~g Stories Involving Recurring Goals, Cognitive Science 2, 235-267 (1978)
- D .Waltz, l.Bogge ss: Visual Analog RepresentatiCin > for Natural language Understanding, 6th-IJCAI, 926-034( 1979)
- S . TsuJI,S.Kuroda and A.Morizono : A Simple CJrloon Fi lm Understanding System, 5th-IJCAI, 609-610 ( 1977)
- S . TsuJI ,M.O sada a'ld M. Yachlda: Three Dimen sional Mov e- ment Analysis of Dynamic Images, 6th-IJCAI , 896-90 I ( 1979)
References were extracted automatically from the PDF and may contain errors.
Full text extracted automatically from the PDF; figures, tables and equations may differ. The PDF is the version of record.
1. Introduction
Recently the research on Artificial Intelligence has become widely practiced, and there have been many papers on story understanding. They are roughly categorized into two classes. The first one correspo~s to researches done by Schank~ Lehnert~ Wilensky and Ca rbonell~ which infer causal relation among things using so-called common-sense such as scripts, goals or human traits etc. They are quite interesting in respect of the points that they contribute to better comprehension of the flow of events in the story. !l,S
The other is on picture processings, its goal is to extract some meaningful actions by observing a sequence of movements in images. This research is also interesting because it may suggest any insight as to how we can recognize higher concept on actions from the view. But its consequence is not necessarily satisfactory as it is difficult to capture action with the help of movements in the scene alone. The reason for this can be well understood considering our cognitive
methodology.
In our life, we utilize many sorts of knowledge sources to capture things surrounding us. We cannot well understand TV -movies without any narration, and an infant child cannot follow the story without pictures. Thus, computer program should positively refer to these sources.
7 Concerning to this point, Waltz has written a paper pointing out that pictorial knowledge often makes it easier to infer the relation not easily deducible from linguistic knowldge . But using the model as a default ono:, it has been not necessarily useful for the recognition of a real situation being :>t hand, because the verification has not been attempted if the predicted things can be well matched to those in the scene .
Seeing these progresses on AI. it is convinced that
- .. visual information is regarded as an essential one for us to capture the complex relations among things with respect to thier locations. For example, when :.Ve ask someone to show the way to station, if he illustrates the way showing a map, it would be easier for us to understand what he says. The computer program should utilize such a pictoral aid concurrently with referring to linguistic knowledges, and then it is easy to analyze the structure of sentences such as those between prepositional groups.
- ·· It is needless to say that the precise comprehension of the meaning of sentences or words with various kinds of sense requires some contextual informations and so-called common-sense knowledge, and there are many sorts of such contextual information. Visual information can also be thought to be a member of such context builders. Considering this :
He takes the apple in the bo~. . . . After reading this fragment alone, 1t IS 1mposs1ble to decide whether "take" means the action "eat" or "get" Even in such a case, the appreciation of informations illustrating the scene leads the sentence analyzer to eas} settlement of such a problem.
2.2 The assumption of the system
- So far, we have written as if all things worked well by taking full advantages of both informations, in reality, however, several assumptions must be set up as in the followings.
- A situation that a line drawing illustrates corresponds to the time when the affairs described in the narration is completed in the aspectual sense.
- As a rule, unless objects are explicitly mentioned in the narration even if they are identifiable in the given scene, the program does not look. for them in the scene and therefore they are not remembered in the forms of assertions as to be in the scene.
Why the maneuver like this is taken here must be described . We have already mentioned the reason in 2.1 from one perspective, that is "interesting". And from another perspective, if such insignificant objects and all the relations among them were memorized in the memory, the combinatorial explosion on the management of such relations would be unavoidable, because a slight change concerning to one of such insignificant things would cause drastic reformation . To put it more concretely, if a person standing in a situation consisting of some rocks moves toward one of them, it is not sufficient to make an alteration only to the relation between the person and the rock. All relations between the person and the other objects in the scene must also be changed, because that person surely approached one of them, but simultaneously went away from the other of them. In case where a situation becomes more complicated, doing such works would make the system unmanageable.
- ( 3) Objects in the scene are looked for in terms· of their colors and relations among their sub-parts. Any rotational movements around any axis is assumed not to happen for the simplicity. The reason. why these assumptions are required is almost clear considering the current state of art on image processing techniques. As this research is an experimental attempt referring to two sorts of knowledge sources, such topics are not taken into considerations.
In the followings, beginning with a data structure used in this system, moving through concrete explanation on programs, and ending with their implementation results.
3. Knowledge representation and data structure
It is well known that know ledge is indispensable to our daily life, but this is also true for a computer when we let him appreciate an environment surrounding him. In this regard, the world knowledge, object models for an object identification, and dictionaries for language analysis are given to this system as its apriori knowledges, which are of vital importance to make inference about a plot of the given story. This reasoning process involves a generation of the description representing their causal chains among events, a construction of the world model in terms of their locations and that of the frame model containing several sorts of properties of objects, and these consequences will be used by our question answering programs mentioned in 5.2. In this section, the data structures will be shown .
3.1 World Knowledge
The implicit knowledge, which we feel we have, explaining various events in the world are built into the system in forms of the semantic network, with a hierarchical structure described in terms of general/specific relations. The network consists of objects or nodes representing their concept and linkages semantically linking them. The node is in the form of frame structure, which have several number of slots for properties or attributes that object will have, and the lowest node in the tree corresponds to the specific object as shown in Fig.!. The linkage between nodes has a label-value pair which illustrates a relation with its score holding between those frames. As is the case with the ordinary frame system, some functions enable our frame to inherit information from the higher frame by following SIS-A relations( see Fig. I). - _ _j~ t .. _ J _ -~ mr.r J lm·tn: J ftlii .\Kt:

3.2 Models for object Inspection
With respect to the inspection of object in a scene, the following two points are assumed.
- (I) We permit the line drawing given as input to have some deep information with respect to the actor's movement toward fore and aft directions in the approximate sense .
- Models are designed to make it easy to describe objects with respect to the mutual relations among their subparts and their colors, and their precise shape is not taken into consideration from the following reason:
When we observe fairly complicated objects in our daily life, we often capture them as the composite set consisting of their elementary subparts, and are not possessed with an awkward idea that they should be matched against their models in a strict sense.
Concerning to the object identification, taking this flexibility i'nto consideration results in a robust program, that is, less subjective to several kinds of noise, and the program can, therefore, recognize what he sees.
A format of the object models is shown in Fig.2. As shown in this figure, the object to be inspected is framed with a rectangle just fitting the boundaries of the object. Next, this rectangle is divided into 9 sub- regions, where each sub-parts locates and what relations it has among others are described in terms of these sub-regions. The assumption that no rotational movement occurs helps to simplify this framing process, as all of its edges are restricted to vertical or horizontal ones.
Fig.J shows a model for desks. At top of it, there is a declaration preceded by the symbol ·PICT, which says that the main property is a DESK, and its subordinative statements, preceded by ·SUBR, follows which means that a breakup operation of it into its subparts.
This breakup process can be repeated so that a hierarchical representation of models are built up. With respect to each statement , the subpart, its color, location and relations among other parts are put in their own places, and yet to each of such statements, a score is appended which is used in a decision whether the matching of object against this model succeeds or not. For an instance, a description (FOOT! COUT ((R) D) 10) means the followings,
- FOOT! is located in a position shown in Fig.J.
- A relation COUT holds between FOOT! and DSK which is its superior constituent of FOOT! , where the relation COUT states that the body of FOOT! is Contignous to that of DSK and that it occupies a region OUTside of DSK.
- ,. If all of these conditions are satisfied, 10 points will be given as its score.


A suitable representaion must be selected so as to make it easy for a program to infere things which might happen and also to make it easy to follow his reasoning when asked some questions. The CD(Conceptual
- dependency) representation exploited by Schank is adopted by the following reasons:
- ( 1) The CD concept leads an evaluator to straightfoward resolution with respect to its interpretaion of ambiguous sense of words.
- The CD makes it easier to reason about the cause-effct relations among events in the story.
In favor of these, the following two dictionaries are used in this system.
3.3.1 Syntactic dictionary
This is a dictionary that is refered by mini-A TN parser which generates a possibility list consisting of all plausible combinations among prepositional groups or fragements of sentence. Lists of ideoms or syntactic roles of words, for some of which the case information, are stored in this dictionary. To date, nearly 200 words have been given their full definitions and more than about 300 words registered into memory in an inperfect form.
3.3.2 CD dictionary
With the help of this dictionary, a surface structure of an input · sentence is translated into the CD representation. Each verb in this dictionary is given a set of rules evaluated by a pattern-directed invocation. For each of these rules, its premises and conclusions are listed in terms of the CD notion, with a demon to be created and procedures to be executed when this rule does a good job. In other words, each rule is in form of so-called production rule and functions are added to it to realize side-effects that is attended by this action. This demon is required to turn many thing into reality, especially to associate assertions believed to be in the cause-effect relation by going through all assertions, or adding some assertions deducible from this rule when neccessary.
3.4 Object Frame
This is a most specific frame placed in the lowest position in the hierarchical tree, and represents an instance of its higher node. This Object Frame (abbreviated as OF here after) has slots containing its properties, its location and all assertions closely related to this object. These slot structures deserve more explanation because of its idiosyncracy.
As a location of an object will be change when the object moves in a scene, its OF must record the current location in a proper form. For this purpose, the slots must be categorized into two classes, one involving constant properties and the other for dynamic attributes such as its location, state, and actions that have something to do with this object. These dynamic properties are managed under the frame number of the scene, that is, the frame number is used as its leading slot. So the slots in an ordinary sense are secondary and can be used as a key to retrieve information, provided that frame number is k.nown(be careful not to confuse the meaning of these two frames).
4. System Organization
A rough sketch of our system is shown in the followings and we want to put them into perspective.
4.1 Syntactic parser
This is a well known ATN parser, which refers to the syntactic dictionary when necessary. It does so-called the syntactic analysis, and on its completion several parsing trees will be generated in the form of possibility list, each of which shows plausible combinations of prepositional groups or others. For the simplicity, we dismiss this and proceed to next.
4.2 Generation of Intermediate notation
After getting parsing trees, they are tested which of them can give the most plausible interpretation for the given sentence by referring to the corresponding image data. To put it more concretly, the verification is conducted to see whether there exist what are expected to be in the scene and also the relations suggested by this possibility list hold among them. For example, let the following fragment given along with Fig.4:
A cat sees the clock above a box above a desk on the chair.
It is clear that image information contributes to exclude interpretations failig to make sense. In this case, at first object inspector searches objects such as chair and desk , because they are probably sitting on the floor . Then it is easy to see that a noun group "the clock" is modified by two prepositional groups of PREPG l, PREPG2 and that PREPG3 functions as an adverbial phrase. A group of lists illustrated below Fig. 4 shows an intermediate representation for this sentence and inner notations concerning to a location, which are recorded into the memory in terms of symbolic expressions because their relations are mentioned explicitly in English.
Note here that if a sentence is so simple that no confusion arises in its analysis, the object inspector can utilize more directly the results from the sentence analyzer. And it should be also noted that the model inspector can use restriction that the case information of words will give to him. Then the remaining objects are found . In this way, once real relations among them are found in the image, they help to decide relation among phrases.
4.3 Object Inspection
As has already been mentioned, in object models their score are recorded along with their structural constraint, . and the effective use of them can enable the pattern matcher to serve as a partial matcher. In this section, how such models are used with the matcher is first shown, and next we provide a simple example to make it easy for readers to be acquainted with matching process.
above a deal< QD the chair.
PREPG2
PREPG3

4.3.1 Utilization of model
Let assume that we are at a verification process if some object can be matched against the model given as in the followings:
( (·SUBR A thsc) <sO>
· Evaluation of this model
Matching is done with respect to some standard reference to the following inequality:
In this expression, scA-i is to be a score a-priori given to this model and xA-i is to be a real score caluculated through the method given as in the followings . .
When this inequality holds, after talung all possible components described below into account, th!s portion of the object is said to be an instance of thts fragment of model. The evaluation of the model itself is also done in a similar way to this method, but in that case the topmost statement of the description will be (·PICT A thsc) .
· Computation of xA-1
(I) For color statement in<Sl>, a color of the object part is checked against the one designated as B in this illustrative model. If they are quite the same one, then set xA-1 to 0, otherwise set it to seA-l. This means that the color information plays a key role in the matching process. .
(2) For relational statement in<S2,S3>, the followtng three components must be taken· into account: . .
[ 1] Verification of C. Note that a descnptto~ wtth respect to C will .have also the same style as wtth the case for A. If the description for C is found successfully by going through the object beings curren~ly inspec.ted, then set yl to a value given by the followtng equatiOn, otherwise set it to scA2.
[2] A contignous relation between A and C must be checked out on this model description. When no discrepancy is found between them, set y2 to 0. If one of their relations is either CIN or COUT and the another is also either of these two specific relations, let y2 be 1!4·scA2. Otherwise, that is, if they are quite unlike each other, set y2 to l/2·scA2.
[ 3] As is the case with [ 2], a locative relatio~ m~st be also looked at. No difference between them tmplles that y3 should be set to 0. If a combination of their positional symbol with respect to A and C is a member
Fig.4 Analysis of prepositional groups.
(CAT! SEE CLOCK!)
((·POS) CAT! ON CHAIR!)
((·POS) CLOCK! ABOVE BOX!)
((·POS) CLOCK! ABOVE DESK!)
of the set ((U,C) (D,C) (L,C) (C,K)), then equate y3 to l/4·scA2, otherwise set it to 1!2·scA2 At last, xA2 is given a value yl+ y2+ y3.
Looking at its matching process in this light, it is easy for readers to understand that a considerably flexible partial matching is possible by fixing its appropriate threshold value. As a more concrete illustration, let consider if an object in Fig.5<a>is identified as the OBAQ, who is a famouse charactor in the Japanese comic strip. As "HAND2" is occluded by something, 10 points will be reduced according to Q) in Flg.5<c>, this operation corresponding to the case [ 1] stated above. Next with respect to [2], 1/4·10 points will be also cut down because a real relation is "CIN" , which corresponds to ® . Note that still more 1/4·10·10/ (10+8) points must be reduced from the total points by taking the effect into consideration which are brought about an evalutation of "MOUTH" , where 114·10 points have been taken off with repect to ® because of the mismatched relation "CI N ". As a result of this evaluation the score for the "BODY" becomes 56.7 points. This means that the score is over the threshold value, then the object in Fig.5<a>can be identified with the "OBAQ" .

4.3.2 ObJect Identification
A process to identify objects assumed to be in the current scene is conducted by first extracting those object explicitly mentioned in the narrations. This indication by utterance means that their descriptions appear in the intermediate representations shown in 4.2. In advance to this execusion, it deserves to think of possibility if the plausible domain for objects can be restricted to a small portion of the image, because there are often cases where objects themselves or things related to them imply their existent environments. If such restrictive conditions are deducible, the object find ing program can search them in this small region, in a top-down fashion . Otherwise, that is, no such condition is found, the program tries to go through a relative narrow space for those objects by assuming that only a little or not at all movements occur from the place they were located in the preceding images, or that animate objects continue moving toward the same direction as in the latest image. If all of those methods fail to find the specified object, and yet it is confirmed that they are in the given scene, the program works on its exhaustive blind search for it . These are the practical use of pictorial constraint implemented in this system, but those with respect to so-called common sense knowledge are, of course, now in effect . The following description for overcoats shows such an example:
OVERCOAT :$$OUTER $$IN-HOUSE
(WEAR ~MAN 10) (IN %DRESSER 8) (ON %CHAIR 5) (WEAR %MAN -5)
A number put in the last position of lists represents the plausibility value for its occurence, where the most plausible one is given 10 points, whereas -10 for the least case as its score. As this description is self-explanatory, nothing might be needed. Along with these maneuvers, it is also important for the program to pay attention to the circumstance around the identified objects in order to find things closely related them, otherwise, the program cannot know a bit but just objects explicitly stated in the narrations, which is too poor. For an example, if some objects are found in the region seemingly corresponding to the sky and yet they cannot be there, the program must infer that they must be supported by something. F·or apples, which appear in our experiment shown later, something might be a tree or supporter like a table.
4.4 World model construction
Each time an image is given to the system, the image processing program records all things identifiable as an object in forms of their coordinate values with respect to a new coordinate system. On completion of these operations for successive frames, it is necessary for him to put those systems together into one global coordinate system as far as possible, in order to capture the global relations among things appearing in the given story. Since those images given along with narrations does not necessarily illustrate consecutive movement, a change in situation, like a case an actor enters a room from outside, often makes it difficult to tidy those systems up into such a unique system. In such a case , all frames following this situation are described with respect to this new system. The appreciation on the depth relations in the images is, in some sense , in effect by providing models with their standard dimensions as a clue to infer their approximate locations.
The procedure to implement the idea is like this: Now assume that the program sees a n-th frame , and its coordinate system is COn, we are working on it with respect to the standard system COA.
- (a) Find an object that appears in both COn and COm (m<n), and yet has a property "still" or "unmovable" in its attribute slot. If such an object can be found, go to (b), otherwise (c) .
- (b) Calculate an origine of the COn with respect to the COA by referring to coordinates of this object in the COn, COm respectively. Then all object believed to be in this frame are put together with respect to COA.
(c) Suspend the current operation and defer this until the position of some objects is determined with respect to COA.
(d) If there occurs a change in the situation, set a global system to this new one, say COA', and a rough relation between the old COA and this COA' is recorded in the COA', if possible.
4.5 Translation Into The CD representation
We show a rough sketch of this translation process by referring to an example. See Fig.6, which is an easy version with the help of English that is equivalent to the content in 3.3.2. Now let assume that ·an intermediate expression (OBAQ TAKE APPLE3) is derived from the procedure given in 4.2. This expression leads the translation process to the execution of rules, R2, R3 and R4 in this order, because their key pattern IP2 successfully matches to this expression. As to the rule R2, it is clear that this is not applicable to the case at hand because a pattern variable %0BJ2, now bounded to APPLE3, does not mean disease. Then the next rule R3 is checked for its legality. Because the value of %0BJ2 is edible, a result in the form of (OBAQ =P= INGEST -0- APPLE3 -D- (MOUTH STOMACH))
is reached from the portion of the rule labeled with 1, if it is comfirmed that the %0BJ2 does not exist anywhere in the scene, which clearly means that it was eaten in this case. Note that the assumption on aspectual sense stated earlier in this paper is used here. The fact that the object is found to be as it was discourages the rule R3 from being attempted, and eventually rule R4 is activated. This is a so-called default rule, so that it concludes the following with no condition
(OBAQ =P= ATRANS -0- APPLE3 -D- (%%% OBAQ)) On completion of execution of each rule, a demon, if attached to the rule, will be invoked in order to inspect causal relations among these CD descriptions asserted so far.
>TAKE
I PI < "08J I VP "OBJ2 TO "PL> ~RI default { -.[3 OIOBJl =P· PTRAftS +-o- "08J2 -D- (""" "PLl -I ll ("08Jl =P· PTRAftS +-o- "08JI -D- (""" "PL>l IIJ IP2 <"OBJI VP "08J2l .. { R2 %0812 means disease ? -r 4 ("0BJ2 ·P· DO: =R· 12) 1 R3 12 <"OBJI ·~= <<PHYS*ST 19J <PHYS*ST.-3)ll { %0812 is edible ? and %0812 is lost -{1 ("08Jl =P· lftCEST -o- "08J2 -D- OIOUTII STOMACH~ { R4 default i 2 ("OBJI ·P· ATRAftS '-Q- "08J2 -D- (""" "08Jill Create causal chain when } 01 %0BJ1 works on %0812
4.8 Answer Generation
By using of informations derived from events stated in the story and things closely related to them, this system can answer some questions concerning to this story.
4.8.1 Query generation
This is easily accomplished by first translating an input sentense into a query by substituting pattern variables for unknown portion of the sentence, as is the case with the interpretation of narrations.
4.8.2 Matching strategy
- A matching procedure used to answer questions differs from the conventional matcher in the following four points.
- It is vital for him to correctly consider what portions of the story are interested in, because the plot proceeds along with time and actors can move around from place to place.
- As precise locative informations concerning to objects in the scenes are not asserted in the forms of CD notation as a rule, a routine that computes them directly from their coordinate values is required.
- With respect to descriptions stating the status of affairs, even if they were asserted in somewhere in the story once and for all, they are there as they have been until they will undergo some changes in their state in
the consecutive frames. Our matcher must take care of this fact.
- Though assertions, in terms of the CD, first represent events that are stated in narrations or those observed in scenes, they are inserted additional relations among events when a reasoning process goes according to the plan that program should assert some facts with respect to causal relations or others. Consequently, when a matcher is going to find expected answers, it must match the query against a proper portion of descriptions by keeping itself free from such clutter.
5. Experimental result
In this section, beginning with the illustration on the detailed process by which representation is built up from the given pictures and narrations, ending with a dialogue referring to its world model constructed through reasoning.
5.1 World model construction
Fig. 7 shows a series of pictures and narrations given to the system. As a result of the story being interpreted, the inner representation as shown in Fig.8, description concerning to Object Frame as in Fig.9 and the world model in regard to the location of Objects as in Fig. IO are obtained. The figure shown in Fig.IO shows a situation corresponding to first half of this story is stuffed into one global system. Note that the dimension of objects are drawn by considering their standard demensions in depth .
- At the first frame, denoted as #FRAME! , program tries to find the OBAQ and APPLEs first comming out into this frame, whereas the ROCK. which isn't stated in words, is not regarded as to be inspected. The tree, however, is successfully recognized because program uses his common-sense knowledge that apples usually grow on trees. That is, in this case, the program first looks for a tree for the purpose of restricting the region the apples would locate, then find them by searching this tree region . From now on, when we refer to the descriptions given in Fig.8, 9, 10, we use the number preceding them.
The floating point number in assertions 3,4,5 shows the position of APPLEs on the TREEl, and these apples are also recognized as a member of the object-group APPLES!, which corresponds to the word "apples" in the narration for the #FRAME!.
At #FRAME2, the assertion 7 are derived through the rule which translate surface representation into the CD form, where "take" are successfully interpreted referring to the relation between STONE! and his HAND. Concurrently with this operation, a demon is activated, shoving a fragment of description having the form of (=R= (12)) in the assertion 7. This is needed to record the fact that the action stated in 7 enables the event in assertion 12 to be caused.
At #FRAME3, a transformational rule required to 'capture the word "throw" is used, and its consequence is correctly predicted and verified, and the representation 12 and a demon D2 are generated_ This D2 convinces the program of the fact that the APPLE3 fell down on the ground in the intervals from #FRAME2 through #FRAME3. ' This also causes a new assertion 16, which states the APPLE3 was fallen with the STONE!, to be gone into the representaion. In addition it to these, creates an another demon which devotes its attention to the APPLE3_for conformation of what might happen to it.





** R)~RESENTATIONS **
He gives OJIRO the apple.
ll 108,\Q =P= PTRANS -o- OBAQ -D- OlPL liiHEREll
2l IOBAQ =P= Pfi'RANS -0- API'LF.51 -R- OUilll OBAQll
3l (($POSl APPLE! ON TREE! +.723:J29E+08 +.264331E··00l
4l (I *I'OSl APPLI!:2 ON TIU:E I + . 21U1823E+00 + . 23B853E+00l
:Jl <<*POSl APPLE3 ON l~EI +.43:J294E+00 +.:J63694E+OOl
61 «*POSl APPLE51 ON TREEll
7) ( 084Q = P= A TRANS - 0- STONE I - D- Olllllll OBAQ) = R= < 12l l
121 <OBAQ =P= SMOVE -0- STONE! -D- (lllllllll ::<I'Lll =E= <71 ·RI= 16 =R= ( 18 23 26 3lll
13) <<*POSl APPLE! ON TIU:El +.71~169E+88 +.271~42E+OOl
141 ( < *I'OSl APPLE2 ON TREt:t + . 26~62:JE+88 +. 245{)96E+80l
l~l ((*POSl APPLES! ON TREEll
161 <OBAQ =P= PROPEL -0- APPLE3 -D- ((+.177382E+03 +.137341E+83 +.977839E+80l
<+ . 196673E+03 +. 173271E+03 +.62060:JE+08ll -1- 121
171 <<*POSl APPJ.E3 CONTACT OBAQ + . I09574E+OI +.7437:JOE+08l
161 <OBAQ =P= ATHANS -0- APPLE3 -D- (lllllllll OHAQl =E= ( 12) =R= 123 26 31ll
19l <<*POSl APPLE! ON TllEEI +.743055E+00 + . 265100E+00l
201 <<*POSl APPLE2 ON TREE! +.312:J00E+00 +.244966E+00l
21 l ( < *POSl APPLE:J I ON TR):Et l
221 <<*PSSl HOUSEl OHAQl
231 IOBAQ =P= Pl~S -0- APPLE3 -D- (lllllllll HOUSEl) -1- 24 =E· ( 12 16ll
241 <OBAQ =P= PTRANS -0- OBAQ -D- (lllllllll HOUSEill
2:Jl <OBAQ =P= PTRANS -0- OBAQ -D- llllllllll I INSIDE HOUSEilll
26l IOBAQ =P= PTRANS -0- APPLE3 -D- llllllllll liiPLl =E= < 12 16ll
271 <<*POSl APPLE3 ON TABLEt +.456522E+00 -.215909E+00l
261 <OJIRO =P· PTRANS -0- OJIRO -D- (lllllllll OBAQl -1- 29l
29l <OBAQ =P= PROPEL - 0- VOICE-D- <OBAQ OJIROll
301 I<*POSl APPLE3 ON TABLEt)
31l <OBAQ =P= ATRANS -0- APPLE3 -R- <OBAQ OJIROl =E= ( 12 16ll
321 <OJJRO =P= INGEST -0- APPLE3 -D- <MOUTH STO~CRll
Fig.8 A portion of primary pass.
| ~~R~BAQ ·~ <•ACT ( t 2) •PLIST <<BODY . 9) <HANOI • II) <HMD2 • HU !HAIR 8R71 (llH3 H2 811 32 31 3011 lEYEI • 121 lBEYEI • 131 lEvt:2 • 141 lBEYE2 .• 1~1 li10UTH . 161 ltrrll • 81181 lllKJlll) 34111 •WHt:RE 120 1201 •SIZE |
| l41 74> •3D (+ . 20~142£+02 + . 131291[+03 + . 182571£+01>> > APPLE~! •• l*ACT 121 •ME~ lAPPLEI APPLE2 APPLE31 oPOS 1611 > APPLE! •• l*�NA~ APPLE~ I •POS 131 *PLIST !lAPP • 31 liiTI Bll21 lllAII 81) l HT2 • BRII ( ( ( A21 71 I I •>1lt:RE 1207 921 •SIZE ll3 211 |
| > -~~P~-~2 203967~:0~ *~":~6!~~~~~7 :r~:~~:~E:;~~~T (( APP • 4> (UTI BR4> ( ((AI) 101) l 111'2 , BR31 l l l A21 91 I) •WHERE l 168 671 •StZF. ( 12 231 •30 (+ . 164348F.+03 + . 8~1006F.+02 +.978261£+08)) |
| > APPLE!! • • l*GNAII APPI.E~I oPOS (51 *PLIST llAPP • 51 lHTI Bl\61 (((All 1211 liiT2 . BMI lllA21 IIIII •WHEIIt: ll83 1301 •SIZE ll2 231 •30 (+ . 179022£+03 + . 13~000E+83 + . 978261~+08)) > TJU:F.I •• l*POS 13 4 3 61 oPLIST llLF.AF • 21 ITillfflK. Ill $liH£.tlf: ( 1~2 61) •SIZE <O:J 1:17) *30 (+ . 134809!:+03 + . :i3H121E+02 + . 882166£.+08)) |
Fig.9 A portion of OF.

.
2 ·COORD <+ . 33437~£+81 + . 272616£+81 +.163016E-8tl *BASE
3 ·COORD 1· . 4293S9E·02 - . 478614~·01 - . 479b~~E-811 ·BASE
4 ·COORD l· . 1112~1E·03 - . 973746E·OI -.118796E·OOI ·BASE
5 ·COORD l · . 3S03:11 E+O:J · . ~H662E·02 ·. 8711296E·OOI ·BASE
6 ·BASE 6 ·COORD lO 0 81 ·IIVLP !INSIDE HOUSEl)
7 *COORD t·.OOOOOOE·OO · . 800000E·88 +.008000E+OOI *RASE 6
a ·coono <+ .ooooooE+oo +.eooooeE+eo +.oeeoooE+oeJ ·BASE 6
Fig. 7 Input scenes and narrations.
I ot:UOIW I 0 0 J .JAS~ I
Fig.lO The positional world model.
At #FRAMES, there is nothing unmovable except for HOUSEl in both #FRAME4 and #FRAMES, but as this HOUSEl is recognized for the first time at #FRAMES, it is used to relate #FRAMES to the global coordinate system denoted as ·BASE in Fig. 10. Here the reader must note that "the house" drawn in #FRAME4 is not perceived to be there as it is not mentioned in the narration at #FRAME4. Consequently it is not found in #FRAME4 untill program look. at things corresponding to HOUSEl at #FRAME4 with utmost care, because HOUSEl is the only object believed to be unmovable at #FRAMES.
At #FRAME6, a change in the situation occurs and a new basic coordinate system is generated, in this case no relation between these basic systems is recorded (see 7-th list in Fig. lO).
Here the TABLE I in 27-th assertion was not mentioned in the narration, but is found to be in a scene, because its existence is predicted and verified through the location of APPLE3 and the fact that the action "put" suggests a place on which the object is placed.
5.2 QA result
This program can answer some questions shown in Fig.ll, where a first few examples are listed along with the reason why the answers can be reached by the program. As with the section S.l, we refer to examples with their number .
- ( 1) The question beginning with "how many" can be answered by counting how many assertions or descriptions match to the query. As to this, the context is recognized as #FRAME4. using "When OBAQ took. the apple", and this context enables the matcher to find a query ((·POS) ?-APPLE ON TREE!) to be the same as the assertions 19. 20.
- By finding that there is a description =E= in the 18-th question, which indicates that the action stated in 12 causes the event in 18 to come about, it is known that the expression 12 gives the answer .
- As with the above example, assertions 23, 26 and 31 are used as an answer simply by finding a descriptor =R= representating causal relations. In this version , how much detail should be given in explanation is not considered very much.
- The 32-th assertion easily leads to this answer. It is easy for a program using the CD representation to match "EAT" to "TAKE", thus we believe that the CD concept fits well the sophisticated knowledge representation requmng many sorts of knowledge sources like our work..
- (S) A frame number is used to answer question . The level of an English generation routine it uses is not so strong yet that some answers are often given in this way.
- There being no assertion concerning to the color of STONE! , this answer is derived directly through seeing a line drawing, itself.
8. Conclusion
The reference on both image and language enables the following effect to be realized.
- An explicit mention of objects can simplify the objects identification process by first restricting the region they will exist to the small one, and by next infering things related to them.
- The confinement of description of objects into the forms of coordinates contributes to a compact representation of events and a combinatorial explosion
: :: WHEN 08!\Q TOOK AN APPLE, HOW I'IANY APPLF.S WERE TIIFII.E Of' Tm : TRt:,: '?<I J
TWO . ::: HOW DID HE T!\KE TilE APPLE ?
HI:: TIIREW THI:: STONt: TO I t :T ·nrt: APPLF. FALL.< 2)
: : : WHY DIll HE '1'/\KF. Tilt: /\I' I' I. F. ?
TO T/\IU: IT TO IllS IIOU>Ot. /\Nil TO PUT IT AND TO GIVt:
IT TO O.J IRO . < 3!
: : : HOW ~!ANY AI'I'LF.S WF.IU: TIIF.I\E ?
THIU:t:.
: : : IIOW M.\NY 1\PI'I.I-:S /\1\F. Tllt:ru: ON Ttn: TIU: t: ?
TWtJ.
:: : IJIIJ O.JII\0 t: /\T Tnt: /\I'I'I.E ? <41
YES · lit: ll I ll .
: : : IUJ\1 1'1.\HY Al'l'l.t:S i\IU: TI!Ent: ?
TWO .
: : : WHEN ll I II m : GFT THE STONE ? < 5 l
AT ... rn:.rn::.! .
: : : WII :\T COI.Oil IS IT ? <6J
IT IS WIIITt:.
: : : WHAT H/\PPENF. II AT 1t'L.AM1·:6 ?
08/\Q t:NTERt:ll TilL HOUSE.
OU/\<l PUT THt: AI'I'L£:.
: : : WJI/\T 11/\Pf't:NJ::Il At'TFR liE TIIIlF.W THE STONE ?
081\(l LF.T TH~: AI'PLt: FALL DOWN .
011/\Q GOT THE APPLE .
OU/\U TOtlK l'IH: 1\PI'I.F. TO TilE HOUSE .
Oll1\<l PUT THE API'LE.
Oll/\<l G/\VE l:JIHO TIIF. API'LF..
Fig.ll QA result.
can be avoided that will occur when all relations among things in scene are asserted in words in the data base .
- ( 3) The relation found in the given scene can he 1 p the interpreter of language resolve interrelation among prepositional groups.
- The usage of production-like representation for the translation from English into the CD description and availabli!ity of pictorial information in this translation makes these operations quite straightforward.
On concerning to (I ),the following must be noted: from the assumption stated first in this paper , there is a case where some objects are not recognized what they are, when they are not stated explicitly and yet not deducible from any facts known so far , even if they are identifiable when compared with their models. This point should be regareded as a good point rather than a defect, because the system at any time can answer questions about such things as the clock. in #FRAME6-8. This clock. was explicitly recognized at any time in the story, but the following question such as "Is there anything on the wall?" or "Is there a clock anywhere?" can be answered by ulitizing the words appearing in these questions as a clue to find and recognize objects. But there is obviously great deal more to be improved in order to completely acheive the goals of this research .
@inproceedings{ijcai1981p18,
title = {A Plot Understanding System on Reference to Both Image and Language},
author = {Abe, Norihiro and Soga, Itsuya and Tsuji, Saburo},
booktitle = {Proceedings of the Seventh International Joint Conference on
Artificial Intelligence, {IJCAI-81}},
publisher = {International Joint Conferences on Artificial Intelligence},
editor = {Patrick J. Hayes},
pages = {77--83},
year = {1981},
month = {8},
note = {Natural Language},
url = {https://ijcai.org/proceedings/1981/18},
}
TY - CPAPER TI - A Plot Understanding System on Reference to Both Image and Language AU - Abe, Norihiro AU - Soga, Itsuya AU - Tsuji, Saburo T2 - Proceedings of the Seventh International Joint Conference on Artificial Intelligence, IJCAI-81 PB - International Joint Conferences on Artificial Intelligence PY - 1981 SP - 77 EP - 83 UR - https://ijcai.org/proceedings/1981/18 ER -Copied
Machine-readable versions of this paper, for tools and AI agents: