Computer Interpretation of PEANUTS Cartoons

Mark R. Adler Department of Artificial Intelli&ence University of Edinbur&h Edinbur&h, Scotland

We describe some features of a pro&ram which interprets cartoons which are two-dimensional representations of events in the three-dimensional world. The main problema confrontin& the reco&nition procedures stem from the difficulties in describin& the irre&ular curved line drawin&s which comprise this domain:

1. Shape Descriptions Curved objects are more difficult to describe than polyhedra. The difficulty is compounded when curved objects are also irre&ular, and vary each time they are hand-drawn.
2. Occlusion Hidden contours cannot bt found by boundary extension for the irre&ular curved cartoon world,

The strate&y we adopt (based on ideas of Guzman [1971]) is to reduct the importance of precise shape descriptions by exploitin& ~ information provided by the use of models.

Individual components of our simple scenes are locally ambi&Uous because each shape can have several possible referents, e.&. a rectan~ular re&ion may be part of a le&, a sock, or a baseball bat, etc. Models are proposed on rather &rosa features and once invoked provide context inforaation to determine the interpretation. The model hierarchy we employ has two main roles:

1. To provide shape information, i.e, a description of the re&ions which occur in the scene at a local level -- ita appearance,
2. To provide information about relationships between re&ions. This is a more global view, It explains how small re&iona fit to&ether to form lar&er, more meanin&ful structures,

The relational models of objects in the Peanuts world &Uidt the a~alysia by applyin& information about how rt&ions may be connected to represent people and objects. Sophisticated techniques are applied to handle complications ariain& from occlusion and variations in orientation. The shape models &uide us in the matchin& of re&ions to shape descriptions. Each shape model has an associated aeries of feature teats (varyin& with the coapltxity of the shape) which are applied to the data definin& the re&ion, Guided by the relational context, the shape testa only need to verify the re&ion identity rather than &enerate

it.

The model information is structured into a hierarchy baaed on functional &roupa of rt&iona,

This deals with self-occlusion as well as with objects which have flexible joints and may appear in various attitudes, These sub-structures are represented by a variety of sub-models, Each reflects a possible re&ion confi&uration accountin& for differences amon& the characters and for the variety of orientations and body positions.

Occlusion

In our cartoon domain, it is important to discover which objects are occluded since this may alter their perceived shape to the point where they can no lon&tr be matched to a shape model . With. proof of occlusion, the system is prepared to lower ita threshold for acceptance; otherwise, the current model must be rejected in favour of a more appropriate one, retainin& previously computed information when possible,

A T-joint may be used as a powerful tool to confirm occlusion, T-joints may be interpreted as the place where an object's boundary disappears beneath another, or else where it reappears. The pairing heuristic finds the correct pair of such T-joints, thus providin& information about rt&ion occlusion. Unfortunately, not all T-jointa indicate occlusion ; they also may occur when one re&ion merely abuts another. The pairin& heuristic is only applied when occlusion is the suspected cause of a re&ion-to-modtl match failure. It can determine the proper pairin& of T-joints to any depth of occlusion. The implemented system also has the capability of dealin& with "curved" T-jointa whose orientation may be ambi&uous.

Experiments and Extensions

We have tested the reliability of the system by systematically diatortin& the presented data. By usin& drawin&s created by interpolation between two characters, we have shown that the shape models baaed on the presence of key features can diatin&uiah between valid representations and "nonsense" scenes. By enlar&in& the heel of a shoe in sta&es, we observed the stron& context effect of the hierarchy that forced acceptance of the distorted shape. Further experiments have led to the refinement of the control structure to use intermediate-level knowled&e to overcome some problems of model instantiation. This work is seen as the first ata&e of a multi-scene analysis, usin& one scene to predict the next,