What is an Image?

1. INTRCDJCTION

AARON is a computer program designed to model some aspects of human art-making behavior, and to produce as a result "freehand" drawings of a highly evocative kind (figs 1,2). This paper describes the program, and offers in its conclusions a number of propositions concerning the nature of evocation and the nature of the transaction -- the making and reading of images in which evocation occurs. Perhaps unexpectedly -- for the program has no access to visual data -- some of these conclusions bear upon the nature of visual representation. This may suggest a view of image-making as a broadly referential activity in which various differentiable modes, including what we call visual representation (note 1) , share a

significant body of common characteristics.

In some respects the methodology used in this '-K>rk relates to the modelling of "expert systems" (note 2) , and it does in fact rely heavily upon my own "expert" knowledge of image-making. But in its motivations it comes closer to research in the computer simulation of cognition. This is one area, I believe, in which the investigator has no choice but to model the human prototype. Art is valuable to human beings by virtue of being made by other human beings, and the question of finding more

efficient modes than those which characterise human performance simply does not arise.

My expertise in the area of image-making rests upon many years of professional activity as an artist-- a painter, to be precise (note 3) and it will be clear that my activities as an artist have continued through my last ten years of '-K>rk in computer-modelling. The motivation for this work has been the desire to understand more about the nature of art-making processes than the making of art itself allows, for urrler normal circumstances the artist provides a near-perfect example of an obviously-present, but virtually inaccessible body of knowledge. The '-K>rk has been informal, and ~ psychology lacks methodological rigor. It 1s to be hoped, however, that the body of highly specialised knowledge brought to bear on an elusive problem

will be some compensation.

MROO is a knowledge-based program, in which knowledge of image-making is represented in rule form. As I have irrlicated I have been my own source of specialised knowledge, and I have served also as my own knowledge-engineer. Before embarking on a detailed account of the program's workings, I will describe in general terms what sort of program it is, and what it purports to do.

First, what it is not. It is not an "artists' tool". I mean that 1t is not interactive, it is not designed to implement key decisions made by the user, and it does not do transformations upon input data. In short, it is not an instrument, in the sense that most computer appl1cat1ons in the arts, and in music particularly, have identified the machine in essentially instrument-like terms.

AARON is not a transformation device. There is no input, no data, upon which transformations could be done: in fact it has no data at all which it does not generate for itself in making its drawings. There is no lexicon of shapes, or parts of shapes, to be put together, assembly line fashion, into a canplete drawing.

It is a complete and functionally independent entity, capable of generating autonomously an endless succession of different drawings (note 4). The program starts each drawing with a clean sheet of paper -- no data -- and generates everything it needs as it goes along, building up as it proceeds an internal representation of what it is doing, which is then used in determining subsequent developments. It is event driven, but in the special sense that the program itself generates

the events which drive it.

It is not a learning program, has no archival memory, is quite simple and not particularly clever. It is able to knock off a pretty good drawing thousarrls, in fact -- but has no critical judgement that would enable it to declare that one of its drawings was "better" than another. That has never been part of the aim. Whether or not it might be possible to demonstrate that the artist moves towards higher goals, and however he might do so through his work, art-making in general lacks clear internal goal-seeking structures. There is no rat10nal way of determining whether a "move" is good or bad the way one might judge a move in a game of chess, and thus no i.rrmediately apparent way to exercise critical judgement in a simulation.

This lack of internal goal-orientation carries with it a number of difficulties for anyone attempting to model art-making processes: for one thing, evaluation of the model must necessarily be informal. In the case of AARON, however, there has been extensive testing. Before describing the testing procedure it will be necessary to say with more care distinguishing here between the program's goals and my own -- what AARON is supposed to do.

Task Definition.

It is not the intent of the AARON model to turn out drawings which are, in oome ill-defined and loosely-understood sense, aesthetically pleasing, though it does in practise turn out pleasing drawings. It is to permit the examination of a particular property of freehand drawing which I will call, in a deliberately general fashion, standing-far­ ness.

The Photographic "Norm"

One of the aims of this paper is to give clearer definition to what may seem intuitively obvious about standing-far-ness, but even at the outset the "intuitively obvious" will need to be treated with some caution. In particular, we should recognise that unguarded assumptions about the nature of "visual" imagery are almost certain to be colored by the XXth century's deep preoccupation with photography as the "normal" image-making mode. The view that a drawn ·image is either:

1. representational ( concerned with the appearance of things) , or
2. an abstraction (i.e. fundamentally appearance-oriented, but transformed in the
3. interest of other aims) or,
4. abstract (i.e. it doesn't stand for anything at all),

betrays just this pro-photographic filtering, and is a long way from the historical truth. There is a great wealth of imagistic material which fits none of these paradigms, and it is by no means clear even that a photograph carries its load of starrling-for-ness ~ virtue of recording the varying light intensit1es of a particular view at a particular moment in time.

It is for this reason that image-making will be discussed here as the set of modes which contains visual representation as one of its members. It 1s also why I used the word "evocative" in the first paragraph rather than "meaningful". My domain of enquiry here is not the way in which particular meanings are transmitted through images and how they are changed in the process, but more generally the nature of image-mediated transactions. What would be the minimum corrlition under which"a set ~ marks rna:( funct1~ asanTmage? This quest1on character1ses economically the scope of the enquiry, and it also says a good deal about how the word "image" i s to be used in this paper, though a more complete definition

must wait until the end.

Art-making a~ Image-making.

The reader may detect some reluctance to say firmly that this research deals with art-making rather than with image-making, or vice-versa. The two are presented as continuous. Art-rna~ is almost always a highly sophist1Cated activity involving the interlocking of complex patterns of belief arrl experience, while in the most general sense of the term ~mage-rna~ appears to be as "natural" as talking. All the same, art-making is a case of image-making, arrl part of what AARON suggests is that art-making rests upon cognitive processes which are absolutely normal arrl perfectly common.

functions which normally require an artist to perform them, arrl thus it requires the whole art-making process to be carried forward as a testing context. The program's output has to be acceptable to a sophisticated audience on the same terms as any other art, implying thereby that it must be seen as original arrl of high quality, not merely as a pastiche of existing work.

A valid testing procedure must therefore contain a sophisticated art-viewing audience, and the informal in situ evaluation of the simulation has been carried out in museum envirorments: the IXX:UMENTA 6 international

Evaluation.

A simulation program rroclels only a small piece of the action, and it requires a context in which to determine whether it functions as one expects that piece to function. AARON is not an artist. It simply takes over some of the exhibition in Kassel, Germany, and the prestigeous Stedelijk Museum in Amsterdam, the two exhibits together running for almost five months and with a total audience of almost half a million museum-goers. In both of these shows drawings were produced continuously on a Tektronix 4014 display terminal arrl also with an unconventional hard-copy device ( to be described later). A PDP 11/34 ran the program in full view of the gallery visitors (fig 3).

In addition and at other times the program's output has been exhibited in a more orthodox mode in mt :;elJIIs and galleries in the US and in Europe.

These exhibits were not set up as scientific experiments. Nor could they have been without distorting the expectations of the au::Hence, and thus the significance of any results. No formal records were kept of the hundreds of conversations which took place between the artist and ment>ers of the ai.Xlience. This report is therefore essentially narrative, but offered with some confidence.

Audience Response.

A virtually universal first asslJIIption of the ai.Xliences was that the drawings they were watching being made by the machine had actually been made in advance by the "real" artist, and someow "fed" to the machine. After it had been explained that this was not the case viewers would talk about the machine as if it were a hlJ!Ian artist. There appeared to be a--general consensus that the machine exhibited a good­ natured and even witty artistic personality, and that its drawings were quite droll (fig 4). Sane of the viewers, who knew my work fran my pre-canputing, European, days claimed that they could "recognise my hand" in the new drawings. This last is particularly interesting, since, while I certainly made use of my own body of knowledge concerning image-making in writing the program, the appearance of my own work never consciously served as a model for what

the program was supposed to do.

~re to the point, while a very small nlJIIber of people insisted that the drawings were nothing but a bunch of randan squiggles, the majority clearly saw them in referential terms. Many would stand for long periods watching, and describing to each other what was being drawn: always in terms of objects in the real world. The drawings seem to be viewed mostly as landscapes inhabited by "creatures", which would be "recognised" as animals, fish, birds and bugs. Occasionally a viewer would "recognise" a landscape, and once the machine's home was identified as San Francisco, since it had just drawn Twin Peaks.

It might be correctly anticipated that on those other occasions when drawings have simply been framed and exhibited without any reference to their origins, the question of their origins

has never arisen, and they have met with a typical cross-section of muselJII-goers responses.

Even without formal evaluation, it might reasonably be claimed that the program provides a convincing simulation of hlJ!Ian performance.

The next part of this paper is divided into five sections. In the first, a general description of the production system ·as a whole is given. The following three sections deal with particular parts of the production system: the ~ CCN'mOL part, the PLANNIOO part, and the part which handles the internal representation of the drawings as they proceed. The second of these, on PLANNIOO, also gives an account of the theoretical basis for the program. The fifth section has something to say about the function of randanness in the program, and also discusses to what extent it might be thought to parallel the use of randanness in hlJIIan art-making behavior. The third and final part draws conclusions.

****************

2 . 'IHE PRO.JRAM "AAOCJN"

2 .1 THE PROOOCTIOO SYSTEM.

---

The main program (note 5) has about three hurored productions. Many of these are to be regarded as micro-productions in the sense that each of them haooles only a small part - an "action-atool" - of a larger conceptual unit of action. FOr example, the drawing of a single l ine, conceptually a single act, actually i nvolves twenty or thirty productions on at least three levels of the system. This fine­ grai n control over the drawing process subscribes both to its generality - most of these action-atoms are invoked by many different situations -- and to its flexibility, since it allows a process to be interrupted at any point for further consideration by higher­

level processes.

Levels of Organisation.

The organisation of the system is heirarchical, i n the sense that the higher levels are responsible for decisions which constrain the domain of action for the lower levels (fig 5). Each level of the system is responsible only for its own domain of decison-making, aoo there i s nc conceptual homunculus sitti~ on the top holding a blueprint and directi~ the whole operation. No si~le part knows what the drawing should turn out to be like. There is some practical advantage to this level-wise splitti~ up of the system, but the program was designed this way primarily for reasons of conceptual clarity, aoo from a desire to have t he program structure itself - as well as the material contained within it -- reflect my uooerstanding of what the human image-maki~ process might be like. I believe that the constant shifting of attention to different l evels of detail and conceptualisation provides this human process with some of its important characteristics . Thus the left part of each product ion searches for combinations of up to five or s i x conditions, aoo each right part may perform an arbitrary number of actions or action-atoms, one of which may involve a jump

to another l evel of the system.

"AR'IIDRK"

The topnost level of the system, the AR'liDR!( level, is responsible for decisions relating to the organisation of the drawi~ as a whole. It dec ides how to start, makes some preliminary decisions which may later determine when and how i t is to finish, and eventually makes that determination. The program currently has no archival memory, aoo begins each drawing as if it has never done one before. (one can easily imagine the addition of a higher level designed to model the cha~es which the human artist deliberately makes in his work from one piece to the next: this level would presumably be called EXHIBITION.)

AR'!IDRK also handles some of the more important aspects of spatial distribution. It is my belief that the power of an image to convince us that it is a representation of some feature of the visual world rests in large part upon the image's fine-grain structure: the degree to which it seems to reflect patterns in the cha~es of information density across the field of vision which the cognitive processes themselves impose upon visual experience.

Put crudely, this means, for example, that a decision on the part of the reader of an image that one set of marks is a detail of another set of marks rather than standing autonanously, is largely a function of such issues as relative scale and proximity. This function is quite apart from the more obviously affective issue of shape ( and hence "semantic") relationship. It is the overall control of the varying density of information in the drawi~, rather than the control of inter-figural relationships, which is handled by AR'!IDRK.

"MAPPIN:i" aoo "PLANNIN:i"

All problems involvi~ the findi~ and allocation of space for the making of individual elements in the drawi~ is handled by MAPPIN:i, though its functions are not always heirarchically higher than those of PLANNING, which is responsible for the development of these individual figures. Sometimes PLANNIN:i may decide on a figure and ask MAPPII'li to provide space, while at other times MAPPING may announce the existence of a space aoo then PLANNING will decide what to do on the basis of its availability. Sometimes, indeed, MAPPING may override a PlANNING decision by announci~ that an appropriate space is not available. A good example of this occurs when P~~Il'li decides to do somethi~ inside an existing closed figure and MAPPING rules that there isn't enough room, or that what there is is t he wro~ shape.

MAPPING will be referred to again in rel ation to the data-structures which consti t ute t he program's internal representation of what it i s doi~, and PLANNII'li also as one of the

centrally important parts of the program.

LINES AND SEC'IORS ----

Below this level the heirarchical structure of the system is fairly straightforward. Each figure is the result of (potentially) several developments, each provided by a return of control to PLANNING. Each of these developments may consist of several lines, and for each of the successive lines of each development of any figure LINES must generate a starting point and an ending point, each having a direction associated with it (fig 6). It also generates a number of parameters on the basis of specifications drawn up in PLANNING which determine how the line is to be drawn: whether reasonably straight, wiggly, or strongly curved, and, if various overlapping modes are called for (fig 7), how they are to be handled.

As I have indicated, lines are not drawn as the result of a single production. When LINES passes control to SEC'IDRS the program does not know exactly where the line will go, since the constraint that it must start and end facing specified directions does not specify a path: there are an indeterminate number of paths which would satisfy the constraint. The program does not choose one, it generates one. SEC'IORS produces a series of "imagined" partial destinations -- signposts, as it were (fig 8) each designed to bring the 1 ine closer to its final end-state. On setting up each of these signposts it passes control to CURVES, whose function is to generate a series of roovernents of the pen which wi 11 make it veer towards, rather than actually to reach, the current signpost. Control is passed back to SEC'IORS when the pen has gone far enough towards the current signpost that it is time to look further ahead, and it is passed back to LINES when the current line has been completed and a new one is demanded by the development still in progress.

We are now down to the lowest level of the program, and to the way in which the curves which make up the drawing are actually generated. This part is not discontinuous from the rest, of course. The flexibility of the program rests in large part upon the fact that the heirarchy of control extends downwards to the finest-grained decisions: no part of the control structure is considered to be so

<Jutomatic that it should fall below the interface line. 'Itlus, the story of how the pen gets moved around follows on from the description of how the intermediate signposts are set up.

Abstract Displays aoo Real Devices

In the earlier versions of the program all the development work was done exclusively on a graphic display, and the "pen" was handled as an abstract, dimensionless entity without real-world constraints upon its movements. Conceptually, however, I always thought of the problem of moving the pen from point A facing direction ~· to point ~ facing direction beta, as be1ng rather like the task of driving acar off a main road into a narrow driveway set at sane known arbitrary angle to it. 'It! is is clearly not a dead-reckoning task for the human driver, but one which involves continuous feedback and a successive-approximation strategy.

It seemed quite reasonable, therefore, to be faced at some point with the problem of constructing an actual vehicle which would carry a real pen aoo make real drawings on real sheets of paper. 'Itlat situation arose in the Fall of '76 when I was preparing to do the museum exhibitions which I mentioned earlier, and decided that if I wanted to make the drawing process visible to a large number of people simultaneously, I would need to use something a good deal bigger than the usual graphic display with its 20-inch screen ·

The Turtle.

The answer turned out to be a small two-wheeled turtle (fig 9), each of its wheels independently driven by a stewing motor, so that the turtle could be steered by stewing the two motors at appropriate rates. It is thus capable of drawing arcs of circles whose radius depends upon the ratio of the two stewing rates.

Since the two wheels can be driven at the sane speed in opposite directions, the turtle can be spun around on the spot and heeded off in a s t raight line, so that this kind of device is capable of stmulating a conventional x-y plotter. But it seemed entirely unreasonable to have built a device which could be driven like a car and then use it to stmulate a plotter. In consequence the pen-driving procedures alrea:iy in the program were re-written to generate the stewing rates for the motors directly - to stay as close as possible to the human model's performance- rather rather than cal culating these rates as a function of decisions alrea:iy made.

The advantage here was a conceptual one, with some practical bonus in the fact that the tur tle does not spend a large part of its ttme spinning instead of drawing. It also turned out unexpectedly that the generating algorithm simplified enormously, and the quality of the freehand simulation tmproved noticeably.

Feedback.

The program does not now seek to any place -­ in cartesian terms - · but concerns itself exclusively with steering: thus the turtle's cartesian position at the end of executing a single conmand is not known in advance. Nor is this calculation necessary when the turtle is oper ating in the real world. It was not designed as a precision drawing device, and since it cannot perform by dead-reckoning for long without accumulating errors, the principle of feedback operation was extended down into this real-world part of the program. The device makes use of a sonar navigation system (fig 10) by means of which the progrCIII keeps traek of where it actually is. Instead of telling it to "go to x,y" as one would tell a conventional plotter, the program tells it "do the following and then say where you are" ·

A more detailed account of the turtle system, and i t's effect upon the s]mulation of freehand drawi ng dynamics, is given in Appendix 1.

1·1 "PLANNI~"

No single level of the progrCIII can be described adequately without reference to the other levels with which it interacts: it has already been mentioned, for exCillple, that MAPPI~ may either precede PLANNI~ in determining what is to be done next, or it may serve PLANNI~ by finding a space specified there. Additionally, any development determined in PLANNI~ may be modified subsequently either as a result of an ilnminent collision with another figure or because provision exists in the program for "stacking" the current development in order to do something not originally envisaged (fig lla,b). All the same, the drawing is conceived predominantly as an agglomeration of figures, and to that extent PLANNI~, which is responsible for the development of individual figures, is of central importance .

Behavioral Protocols in Image-Making.

Of the entire program, it is al so the par t least obviously related to the effects which it accomplishes. While the formal results of its actions are clear enough -- an action calling for the closure of a shape will cause it to close, for exampl e - it is not at all clear why those actions result in the specifically evocative quality which the viewer experiences.

A rule-by rule account of this effect is not appropriate, because the individual rules do no more than implement conceptual entities -­ which I will call behavioral pcotocols

- which are the fundamental units from which the program is built. These protocols are never explicitly stated in the program, but their existence is what authorises the rules. Thus, before describing in detail what is in PLANNING I should give an account of the thinking which preceeded the writing of the program, and try to make clear what I mean by a protocol.

Background.

It is a matter of fact that by far the greatest part of all the imagery to which we attach the name of "art" canes to us from cultures more or less remote from our own. It is also a matter of fact that within our own culture, and in relation to its recent past, our understanding of imagery rest to a great extent upon prior common understandings, prior cultural agreements, as to what is to stand for what -­ pr ior, that is to say, to the viewing of any particular image. It is unlikely that a Renaissance depiction of the Crucifixion (~of Christ" being understood here by means of just such an agreement!) would carry any great weight of meaning if we were · not already familiar both with the story of Christ and with the established conventions for dealing wi th the various parts of the story. Indeed , we might be quite confused to find a depiction of a beardless, curly-headed youth on the cross unless we happened to possess the now-abstruse knowledge that Christ was depicted that way - attaching a new set of meanings to . the old convention for the representation of Dionysus - until well into the 7th century. In general, we are no longer party to the agreements which make this form acceptable and understandable . We must evidently distinguish between what is understandable without abstruse knowledge -- we can, indeed, recogmse the figure on the cross ~ a figure -- and what is understandable only by virtue of such knowledge.

In the most general sense, all cultural conditions are remote from us, and differ only in the degree of their remoteness. We cannot really comprehend why the Egyptians made sphinxes, what Michelangelo thought the ancient world had in common with Christianity, or how the internal combustion engine was viewed by the Italian Futurists seventy years ago who wanted to tear down the musell!ls in its honor. What abstruse knowledge we can gain by reading Michelangelo's writings, or the Futurist Manifesto, does not place us into the cultural environment in which the ...ark is embedded. A

cul ture is a conti1uum, not a static event: its understandings and meanings shift constantly, and their survival may appear without close scrutiny to be largely arbitrary. In the extreme case, we find ourselves surrounded by the work of earlier peoples so utterly remote from us that we cannot pretend to know anything about the people themselves, much less about the meanings and purposes of their surviving images.

The Paradox of Insistent Meaningfulness.

There is an Unplicit paradox in the fact that we persist in regarding as meaningful -- not on the basis of careful and scholarly detective work, but on a more directly confrontational bas is -- Unages whose original meanings we cannot possibly know, including many that bear no explicitly visual resemblance to the things in the wbrld. Presumably this state of affairs ar ises in part from a fundamental cultural egocentrism -- what, we ask, ...auld~ have i ntended by this Unage arrl the act of making it? -- which is fundamentally distortive. There has also been a particular confusion in this century through the widespread acceptance of what we might call the telec<mnunications model of our transactions through unagery, par tlcularly since in applying that model no dif ferentiation has been observed between the culture we live in and the cultures of the remote past. In the view of this model, original meanings have been encoded in the Unage , and the appearance of the Unage in the ...or ld effects the transmission of the meanings. Allowing for noise in the system -- the i nevitability of which gives rise to the not ion, in art theory, of "interpretation" -­ the reception and decoding of the Unage makes

the original meanings available.

However useful the model is as a basis for examining real telecommunication-like situations, in which the intended meanings and their transformations can be known and tracked, i t provides a general account of our t ransactions through Unages which is quite inadequate. The encoding and decoding of messages requires access to the same code-book by both the Unage-maker and the Unage-reader, and that code-book is precisely what is ~ carried across from one culture to another.

I think it is clear also that the paradox of insistent meaningfulness, as we might call it,

cons t 1tutes the normal condition of Unagemediated transactions, not an abnormal condition. It evidently exterrls below the level at which we can recognise the figure, but not what the figure stands for, since so much of the available imagery is not in any very obvious sense "representational" at all. The paradox is enacted every time we look at a few marks on a scrap of paper arrl proclaUn them to be a face, when we know perfectly well that they are nothing of the sort.

Cognitive Bases for ~ Structure.

In short, my tentative hypothesis in starting work on AARON was that all Unage-making and all image-reading is mediated by cognitive processes of a rather low-level kind, presumably processes by means of which we are able to cope also with the real ...orld. In the absence of common cultural agreements these cognitive processes ...auld still unite Unage­ maker and Unage-viewer in a single transaction. On this level -- but not on the more complex culture-bound level of specific iconological intentionality -- the viewer's egocentricity might be justified, since he could correctly identify cognitive processes of a familiar kirrl in the making of the Unage. But let me detail this position with some care. I am not proposing that these processes make it p->ssibl e for us to understand the interrled meanings of some remotely-generated Unage: I am proposi ng that the interrled meanings of the maker play only a relatively small part in the sense of meaningfulness. That sense of meaningfulness is generated for us by the structure of the Unage

rather than by its content.

I hope I may be excused for dealing in so abbreviated a fashion with issues which are a good deal less than self-evident. The notion of non-enculturated behavior -- arrl that notion lurks behind the last few paragraphs, obviously -- is a suspect one, since all human behavior is encul turated to some degree: but my purpose was not to say what part of human behavior is dependent upon enculturating processes and what is not. It was sUnply to identify some of the determinants to a general Unage-structure which could be seen to be common to a wide range of enculturating patterns. The Unplication seemed strong -- and still does -- that the minUnum condition for generating a sense of meaningfulness did not need to include t he assumption of an intent to c<mnunicate: that the exercise of an appropriate set of these cognitive processes ...auld itself be sufficient

to generate a sense of meaningfulness.

Cognitive Skills.

The task then was to define a suitable set. I have no doubt that the options are wide, and that my own choices are not exclusive. I chose at the outset to include:

1. the ability to differentiate between figure and ground,
2. the ability to differentiate between open and closed forms, and
3. the ability to differentiate between insi.deness and outsideness (note 6) ·

AARON has developed a good deal from that starting point, and some of its current abilities clearly reflect highly enculturated patterns of behavior. For example, the program is now able to shade figures in a mode distinctly linked ~Renaissance and post­ Renaissance representational modes: other cultures have not concerned thenselves with the fall of light on the surfaces of objects in the same way. Nevertheless, a large part of the program is involved still in demonstrating its awareness of the more primitive differentiations.

Protocols and Rules. ---

Against this backgrourrl, I use the term ~otocol to mean the procedural instantiation o a formal awareness. This is clearly a deflLnit1on wh1ch rests upon cognitive, rather than perceptual, modes, since it involves the awareness of relational structures. Thus, for example, the program's ability to differentiate between form and grourrl makes possible an awareness of the spatial relationships between forms, and generates finally a set of avoidance protocols, the function of which is to proh1b1t the program from ignoring the existence of one figure in drawing another one. The protocols thenselves are not explicitly present in the program, and are manifested only through their enactment by the rules which describe what to do in particular circumstances where the overlapping of figures is threatened.

Figure Development

In keeping with the heirarchical structuring which informs the progrCIII as a whole, PLANNit'li considers a figure to be the result of a number of developments, each determined in part by what has gone before. The program enacts a number of reeetition protocols, and a single development 1n the making of a figure can often involve the repetition of a single action (fig

12) , rather than the agglomeration of different actions. The first productions to deal with the first development of any figure decide, on the basis of frequency considerations, that this figure will be closed, that it will be open, or that it will be, for the moment, "unCOI11llitted" that is, a line or a complex of lines will be drawn, but only at . a later stcqe will it be decided whether or not to close. If the primary decision is for closure,

then PLANNING will decide between a number of opt ions, 100s tl y having to do with size and shape MAPPING permitting -- and with configuration. In same cases it will not actually draw the boundary of a closed form at all, and will leave the definition of the occupied space to await subsequent spac~ filling moves.

If the decision is for a non-closed form, then again a ntmlber of options are open. In both cases the available options are stated largely in terms of repetition protocols, the enactment of which determines the formal characteristics of the resulting configuration. These characteristics are not uniquely defining, however, and a number of different formal sub­ groups may result from a single repetition protocol and its rules. For example, one such protocol, involving a single line in this case, requires the line to move a given distance (IOOr~or-less) and then change direction, continuing this cycle a given number of times. All the figures marked in (fig 13) result from this: the details of implenentation in the individual cases are responsive to their unique environmental conditions, and in any case may be changed at any point by the overriding avoidance protocol, which guarantees the territorial integrity of existing figures.

Thus the program will know at the beginning of each development what the current intention is, but will not know what shape will result. A closed form generated by a "go,turn, repeat" cycle may in fact turn out to be extremely long and narrow (fig 14), and a mrnber of secorrl developments associated with a closed-form first development will then be unavailable: there will be a limit, for example, upon what can be drawn inside it, though it may develop in other ways, as this one does.

Proliferation.

Even with constraints of this sort there is a signi ficant proliferation in the number of productions associated with the secorrl devel opment of any figure. A typical first development might be initiated by:

If (this is a first development - arrl the last figure was open and at least n figures have been done and at least q of them were open

and at least t units of space are now available)

Then

Th1s figure will be closed

specifications for repetition

specifications for configuration

to rmve on from this point:

If (this is a secorrl development

- arrl the first was closed

and its properties were

a. (size)

b. (proportions)

c. (canplexity)

d. (proximity to ··· )

Then either

-- --1-.-divide it

• ··· specifications or

2. shade it

··· specifications ···

or

3. add a closed form to it ··· specifications ···

4. do a closed form inside

·.· specifications ···

or

5. do an open form inside ··. specifications ···

This is a prototype for an expanding class of productions, each responding to a different combination of properties in the first development. Similarly, continuation will requi re .··

If (this is a third development -- and the first was a closed form properties ··. and the second was a c 1 osed form properties .·. )

Then

shade the entire figure: specification 1: or specification 2: or specification 3: a penumbra.

a boulder with a hole in it

a flat shape with a hole

If (this is a third development and the first was closed and the second was a series of parallel lines inside it ··· and the remaining inside space is at least s .·. )

Then

do another series of lines: specification 1: perpendicular to first ··· or specification 2: alongside the first ···

figure 15.

or specification 3: do a closed form in available space ·.·

(note 7).

The Relationship of Closed Forms and ~ Forms.

The same proliferation of options occurs for open-lined structures also, but not to the same degree. One of the interesting things to cane out of this program is the fact that open-line structures appear to function quite differently when they are alone in an Unage than when they appear in the presence of closed forms. There seems to be no doubt that closed forms exert a special authority in an image -- perhaps because they appear to refer to objects -- and in their presence open-lined structures which in other circumstances might exert sUnilar pressure on the viewer are relegated to a sort of spatial connective-tissue function. A sUnilar context-dependancy is manifested when material is presented inside a closed form (fig 15): it is "crlopted", and becomes either a detail of the form, or markings upon it. This seems to depend upon particular configurational issues, and especially the scale relationship between the "parent" form and the newly introduced material. This manifestation is Unportant, I believe, in understanding why we are able to recognise as "faces" so wide a range of closed forms with an equally wide range of internal markings following only a very loose distribution specification.

Limits on Development.

At the present time no figure in the program goes beyond three developnents, and few go that far, for a number of reasons. In the first place, most of the (formal) behavior patterns in the program were initially intended to model a quite primitive level of cognitive performance, and for most of these a single developnent is actually adequate. Once a zig­ zag line has been generated, repetition, for example -- as it is found in existing primitive models -- seems limited to those shown in (fig 16) ·

It has remained quite difficult to cane up with new material general enough for the purposes of t he program. It is the generality of the protocols which guarantees the generality of t he whole, and new material i.s initiated by the introduction of new protocols. On the level of the procedures which carry out the action parts of the subsequently-developed productions, the approach has been to avoid accumulation of special routines to do special things. There is only one single procedure adapting the prococo1s of repetition and reversal to the generation of a range of zigzag-like forms, for example (fig 13).

But there has been another, and equally significant reason, for the 1 imitation upon permi ssible developnents. It is the lack of ooequate, aoo adequately important, differentiations in the existing figures. For the primitive model represented by the earlier states of the program it was almost enough to have a set of abilities called up by the most perfunctory consideration of the current state of the drawing: the stress was on the definition of a suitable set of abilities (as represented by the right-hand parts of the productions), and as it turned out it was quite difficult to exercise those abilities without generating moderately interesting results. But for a more sophisticated model it is clearly not enough merely to extend that set of abilities, and the problem of determining why the program should do this rather than that becanes more pressing.

The 1 imitation here can be considered in two ways. One is that I had reached the point of exhausting temporarily my own insights into the image-building process. The other is that I hoo not mooe provision in the first versions of the program for being able to recognise the kioo of differentiations I would want to deal with -­ since I could not know at the outset what they were going to be -- and thus lacked a structure for developing new insights. This leads to a consideration of my next topic: how the program builds its own representation of what it has done up to any point in the making of the drawing.

..£ ._! INTERNAL REPRESENTATIOO

In the earlier stages of the development of the program, provision had been made for progressive access to the information stored in the data-structure, following the principal that it should not have to access more than it actually needed for the making of any particular decision. In practise, a great deal more was stored than was ever accessed. At the first level of detail the program made use of a quite coarse matrix representation, in each cell of which was stored an identifier for the figure which occupied it, and a number of codes which designated the various events which might have occurred in it: a line belonging to a closed form, a line belonging to an open form, a line junction, an unused space inside a closed figure, and so on. Obviously, it was not possible to record a great deal in this way, and data concerning the connectivity of the figure in particular required a second level of the structure.

This was an unpleasantly elaborate linked-list structure of an orthodox kind. By definition, the kind of drawing MOON makes is not merely a growing, but a continuously-changing, structure. What was a point on a line becanes a node when another line intersects it, and this change has to be recorded by updating the existing structure, which must now ideally show the four paths connecting this node to four

adjacent nodes.

Both updating this structure and accessing the information contained within it proved to be quite tiresome, and the scheme was never general enough to admit of further development. As a result, it was used less and less, and decision -making has been based almost exclusively on the information contained in the matrix on the one hand, and in a third level of the structure, a simple property-list attaching to each figure, on the other. The most surprising thing about this simplistic and distinctly ad-hoc scheme is that it was actually quite adequate to the needs of the program.

Explicit Data am Implicit Data.

Human beings presumably get first-order information about a picture by looking at the picture. I have always found it quite frustrating that the program couldn't do the same thing: oot because it made any difference to the program, but because it made it difficult for me to think about the kind of

issues I believed to be significant. Part of the problem of using a linked-list structure to represent the connectivity of a figure, for example, " derived fran the fact that connectivity had to be explicitly recorded as it happened: it would have been much too difficult to traverse a structure of this kind post-hoc in order to discover facts about connectivity. If one could traverse the figure the way the eye does -- loosely speaking! -- it would not be necessary to give · so much attention to recording explicitly all the data in the world without regard for whether it would ever be looked at again.

In short, the primary decision to be made was whether to accept the absolute non-similarity of picture and representation as given, devise a more sophisticated list-structure and drop the matrix representation altogether, or to drop the list-structure and develop the matrix representation to the point where it could be very easily traversed to generate information which was implicit within it. I opted for the latter. A description is included in Appendix 2, though at the time of writing (December '78) the implementation is not yet canplete.

..£ · 2_ 'mE Ft.JOCTIOO OF RANOOMNESS ·

This section does not deal with any single part of MRCN: randanness is an active decision­ making principle throughout the program, and I think it is important to say why that is the case. As a preface, it might be worth recording that beyond the limits of a mathematically sophisticated community most people evidently view randomness in a thoroughly absolutist fashion, and as the opposite to an equally absolute determinisn. There is a firmly-held pop.Jlar belief that a machine either does exact 1 y what it has been prog r CIIITled to do, or it acts "randanly". The fact that AARON produces non-randan drawings, which its progranmer has never seen, has given many people a good deal of trouble.

What I mean by "randanness" is the impossibility of predicting the outcome of a choice on the basis of previously-made choices. It follows, of course, that "randanness", in this sense, can never be absolute: if the danain of choice is the set of positive integers, one must be able to predict that the outcane will be a positive integer, not a cow or a color. In AARON the danain of choice is always a great deal more constrained than that, however. The corollary to the notion of randanness as a decision-making principle is the precise delineation of the choice space: in practice, the introduction into the program of a new decision characteristically involves the setting of rather wide ltmits, which are then gradually brought in until the range is quite small.

RaOOomness ~ Design am ~ Default.

AI researchers in more demonstrably goal­ oriented fields of intellectual activity must obviously spend much ttme and effort in trying to bring to the surface performance rules which the expert must surely have, since he performs so well , I amoot in a position to know to what extent "Let's try x" would constitute a powerful rule in other activities: I am convinced that it is a very powerful rule i ndeed in art-making, and more generally in what we call creative behavior, provided that "x" is a ment>er of a rigorously constrained set.

A nunber ·of arti.sts in this century - perhaps roore in music than in the visual arts - have deliberately and consciously employed randomising procedures: tossing coins, rolling dice, disposing the parts of a sculpture by throwing them on the floor, and so on. But this stmply derives a strategy from a principle, am examples of both can be found at almost any point in history. It is almost a truism in the trade that great colorists use dirty brushes. Leonardo recommended that the difficulty of s tarting a new painting on a clean panel -­ every painter knows how hard that first mark is to make -- could be overcome by throwing a di rty sponge at it (note 8). But one suspects t hat Leonardo got to be pretty good with the sponge! An artist 1 ike Rubens would htmself only paint the heads and hands in his figure compositions, leaving the clothing to one assistant, the lamscape to another, and so on. All the assistants were highly-qualified artists in their own right, however. The process was not unlike the workings of a modern film crew: the delegation of responsibility reduces the director's direct control, am randamises the tmplementation of his i ntentions, while the expertise and commonly­ held concerns of the crew provide the ,}tmits (note 9).

Random ising rules. in the Program: Rules and Meta-

_____

For the human artist, then, randamising is not unconstrained, and therefore cannot be characterised by the rule "If you don't know what to do, do anything". Rather, one suspects the existence of a meta-rule which says,

"precisely define a space within which any choice will do exactly as well as any other choice". In ~. the tmplementation of the low-order rule has the following form:

If (a and b and ··· n) Then p% of the tUne do (x): q% of the time do (y): r% of the ttme do (z):

which fills out the description of the format discussed in PLANNitli. The sane frequency­ controlled format is used within the action part of a production in determining specifications:

make a closed loop: specification 1: number of sides 50% of the ttme, 2 sides (simple loop) 32% of the ttme, 3 sides specification 2: proportion 50% of the time, between 1:4 and 1:6 12% of the ttme, between 3:4 and 7:8 specification 3.

AARON has only the simplest form of these meta-rules, which are used to determine the bounds of the choice space:

if(a) lowbound is La, highbound is Ha if(b) lowbound is Lb, highbound is Hb if(n) lowbound is Ln, highbound is Hn specification taken randomly between lowbound and higbound

where a,b,n are varying conditions in the state of the drawing. No consistent attempt has been made to develop more sophisticated meta-rules. In the final analysis, the existence of such rules tmplies a judgemental view of the task at hand, and they are consequently beyond the scope of a program like ~. which is not a learning program and has no idea whether it is doing well or badly.

The Value of Randomness.

----

What does randomness do for the image-maker? Primarily, I believe its function is to produce proliferation of the decision space without requiring the artist to "invent" constantly. One result of that function is obviously the generation of a much greater number of discreet

terminations than would otherwise be possible, and consequently the sense that the rule-set is a great deal more complex than is actually the case. A second result is that the artist faces himself constantly with unfamiliar situations rather than following the sane path unendingly, and is obliged to pay more attention, to ....urk harder to resolve unanticipated juxtapositions. It is a device for enforcing his own heightened participation in the generating process.

This last might seem less ilnportant in AARON: the program's attention is absolute, after all. But for the viewer the fact that AARON exercises the function is quite important. There is one level of our transactions with images on which we respond with some astuteness to what is actually there. The fact that AARON literally makes decisions every few microseconds -- not binary decisions only, but also concerning quantitative specifications -­ shows clearly in the continuously changing direction of the line, in every nuance of shape, and succeeds in convincing the viewer that the~e is, indeed, an intelligent process at work behind the making of the drawings.

················

3 · CCNCWS IONS =z::============

AARON produces drawings of an evocative kind. It does so without user intervention: without recourse to user-provided data: and without the repertoire of transformational manipulations normal to "computer graphics". It remains now, if not to propose a coherent theory of image­ making, at least to pull together those fragments of explanation already given into something resembling a plausible account of why AAROO works.

This wi 11 be largely a matter of putting things in the right places.

Art-making and ~making

First: no adequate justification has yet been given for the many references to art and art­ making, dS opposed to images and image-making, beyond saying that the first are a special case of the second. What makes them special?

Art is a bit like truth. Every culture has, and acts out, the conviction that truth and art exist, no two cultures will necessarily agree about wPat they are. There is no doubt, for example, that we use the word "art" to denote activities in other cultures quite unlike what our own artists do today, for the quite inadequate reason that those earlier acts have resulted in objects which we choose to regard as art objects. If it is surprisingly difficult to say what art is, it is not only because it is never the sane for very long, but also because we evidently have no choice but to say what it is for us.

All the same, no justification is possible for making reference to it without attempting to say - once again! -- what it is, and doing so in terms general enough to cover the greatest m.mber of examples. Also, those terms should do something to account for the extraordinary persistence of the idea of art, which transcends all of its many examples.

Briefly, my view is that this persistence stems from a persistent and fundamental aspect of the mind itself. It would be stating the obvious here to propose that the mind may be regarded as a symbol processor of power and flexibility. I will propose, rather, to regard it as devoted primarily to establishing symbolic relationships: to attaching significance to events, and asserting that this stands for that. This is, surely, a large part of what we mean by understanding.

As for art: in its specifically cultural aspects art externalises specific assertions-­ the mmber three stands for the perfection of God, the racing car stands for the spirit of roodern man, the swastika stands for the semi­ mythical migrations of the Hopi people, or for a nl.lllber of other things in a nl.lllber of other cultures. But on a deeper level, art is an elaborate and sophisticated game played around the curious fact that within the mind things can stand for other things. ~t is almost always characterised by a deep preoccupation with the structures of standing-far-ness, and a fascination with the apparently endless diversity of which those structures are capable. What we see in the muselltls results from a complex interweaving of the highly j.ndividuated and the highly enculturated, and in consequence any single manifestation is bound firmly to the culture within which it was generated: or it is rehabilitated to serve new ends in a new culture. But ultimately, art itself, as opposed to its manifestations, is universal because it is a celebration of the

hlltlan mind itself.

The Embeddedness of Knowledge

Second: much of what has come out of the wri ting of AARON has to be regarded simply as extensions to the body of knowledge which the program was intended to external ise. \'lriting it was not merely a demonstrative undertaking, and i t is far from clear what ·has been raised to the surface and what newly discovered. I have regarded the program as an investigative tool, though for present purposes the distinction is not important.

It remains impossible to give an adequate account of this knowledge other than by reference to the program itself. There are several reasons for this. In the first place, this knowledge does not present itself initially as predominantly prescriptive. The fi rst intuition of its existence comes in the form of an awareness that an issue -- closure, repetition, spatial d1stnbution is s ignificant: the program should be structured in terms of that issue, as well as in terms of all the other issues already contained. In this sense the left parts of the productions might eventually be taken together to represent the set of issues which AARON believes to be worth attending to in the making of an image. But this stage comes much later, and by this time an individual production functions as part of a fabric of issues, with so many threads tying it to so many knowledge sources, that a one-to-one account of how it achieves its effect is generally out of the question.

In fact, there is only a single example I can call to mind in which an effect can be ascribed with certainty to a single production: a par ticular class of junction in a meandering horizontal line will infallibly generate strong l andscape reference, though only if the

branching at the junction goes off on the lower side of the line (fig 17). This degree of specificity is certainly exceptional, but less powerful as an evocator rather than more so .

In general, this particular class of junction it is more easily characterised visually than verbally -- tends strongly to denote spatial overlap: but the specific effect is evidently quite context-dependant, and dependant also upon the pr~cise configuration of the junction itself. '

"Personality" as ~ Function of Complexity.

At the higher end of the scale of effects, the problem of saying what causes what becomes more difficult still. I have never been able to understand how there can be such general agreement about the "personality" which AARON's drawings project, or why that "personality" ar:pears to be 1 ike my own in a number of respects. Personality has never been an issue on the conscious level of writing code, and I know of nothing in the program to account for it. 'Ib put the problem another way, I would not know how to go about changing the program to project a different "personality".

I assume that the personality projected by an image is simply a part of a continuous spectrum of projection, not distinguishable in type from any other part. But I am forced l'lC'M to the conclusion that these more elusive elements of evocation-- personality is only one of them, presumably are generated out of the complexity of the program as a whole, and not from the action of program-parts: that given an adequate level of complexity any program wi ll develop a "personality". This "personality" may be more or less clear in individual cases, and may perhaps depend upon how many people have worked on the program -- AARON is almost exclusively my own work- but it will in any case be a function of the program, and outside the willful control of the progrcmner. If this is the case it seems extremely unlikely that any complete causal account of the workings of a program would ever be possible.

The Continuousness of Image-making and Image-read log.

Third: I want to return to the question which lies at the root of this work. What constitutes a minimum condition under which a set of marks will function as an image?

The reader will have noted that much of what has been written here appears to bear as much upon the business of image-reading as i t does upon image-making. There is no contradiction: the central issue being addressed is the image-mediated transaction itself, and image­ maki ng in particular has no meaningful, or examinable, existence outside of that transaction. Knowledge about image-making is knowledge about image-reading: both rest upon the sane cognitive processes. Thus the skilled artist does not need to enquire what the viewer sees in his work: the satisfaction of his own requirenents guarantees it a reading in the ll«'lrld, and the explicit iooividual readings which it will have are irrelevant to him. The trainee artist, the student, on the other haoo, frequently respooos to his teacher's reading of his work by objecting, "You're not supposed to see it that way", evidently unaware that the reading does not yield to conscious control. Lack of skill in image-making more often than not involves a failure to discern the difference between what is in the image-maker's mind and what he has actually put on the canvas.

It is equally true, I believe, that image­ reading has no meaningful existence outside the transactional context: not because the whole event is always present -- it almost never is - but because every act of image-reading is initiated by the unspoken assertion "What I see is the result of a willful hlJTian act". That is a part of what we mean by the word "image". However much we may amuse ourselves seeing dinosaurs in clouds or dragons in the fireplace, we have no difficulty in differentiating between marks aoo shapes made by man, and marks aoo shapes made by nature, and we do not hesitate to assign meaning in the one case where we deny it in the other: unless we belong to a culture with a more animistic attitude to nature than this one has.

In short, I believe that the first requirement of the coooition in the question is the undenied assumption of hliTian will (note 10).

The rest of the coooition is given by the display of behavior which draws attention to a particular group of cognitive elements. In other words, evidence of cognitive process may be substituted for the results of an act of cognitiun. An actual desire to communicate -­ which may include the simple desire to record the awearance of the world - is not a

necessary condition.

AAroN's strength lies in the fact that it is designed to operate within, and feed into, the transactional context, not to reproduce the aesthetic qualities of existing art objects. It takes full advantage of the viewers' predispositions aoo does nothing to disabuse them: iooeed, it might fairly be jooged that some part~ ·>f the program -- the simulation of freehaoo dynamics, for example - are aimed primarily at sustaining ~ illusion (note 11).

But the illusion can only be sustained fully by satisfying the coooitions given above, and once that is accanpl ished the transactions which its drawings generate are real, not illusory. Like its hlJTian counterpart, AAroN succeeds in delineating a meaning-space for the viewer, and as in any normal transaction not totally prescribed by prior cultural agreements, the viewer provides plausible meanings.

Standing-far-ness.

Fourthly: there is a multitude of ways in which something can staoo for sane thing else, and in adopting the general term "staooing-for-ness" I inteooed for the moment to avoid the excess meanings which cling to words like "symbol", "refer rent", "metaphor", "sign", and so on: words which abound in art theory and art history. An image, I have said, is something which staoos for something else, and of course it is quite plain that I have been discussing only a very small subset of such things.

What are the defin5~ characteristics of this sooset?

Before attempting to answer that question, it should be noted that, while MRCN' s performance is based upon vision-specific cognitive modes (note 12), there are two closely related questions which cannot be asked about AARON at all.

Images of the WOrld aoo its Objects.

The first of these has to do with the fact that in the real world people make images of things.

How do people decide what marks to make in relation to those things?

It is difficult to avoid the conclusion that image-making as a whole is vision-based, even though it bears directly on the issue of appearances only occasionally. It is my belief that even when an image is not purposively referential -- as is the case with AARON -- or when the artist seeks to refer to some element of experience which has no visual counterpart, it is his ability to echo the structure of visual experience which gives the unage its plausibility (note 13).

The Persistence of Motifs

The secooo question has to do with the fact that actual image elements, motifs, have been used over aoo over aga1n throughout hlJTian history, awearing in totally disconnected cultural settings, aoo bearing quite different meanings as they do so. What is it that makes the zigzag, the cross, the swastika, squares, triangles, spirals, mandalas, parallel lines, combs (fig 18), ubiquitous, so desirable as imagistic raw material?

My own answer to this question is that the cognitive modes and their dependant behavioral protocols are absolutely ubiquitous, and that the recurring appearance of these motifs is

hardly even surprising (note 14). In fact, we have only to start cataloguing the motifs to realize that most of them are simply formed through the combination of simple procedures. The swastika, for example, is both cross and zigzag, just as the mandala is cross and closed form, and the so-called diamond-backed rattlesnake motif of the Californian Indians is a syrrmetrically repeated zigzag (fig 19).

Taken together, these two questions point to the dualistic nature of image-making. If, as I believe to be the case, it can be shown that the representation of the world and its objects by means of images follows the same cognition­ bound procedures as the simpler images I have been discussing, then it will be clear that the form of an image is a function both of what is presented to the eye and of the possession of appropriate modes. -

Representation.

I said at the outset that my conclusions would bear upon the nature of visual representation, as distinct from what the AI/Cognitive Science ccmnunity means by the word "representation". It is still the case that my specific concerns are with what people do when they make marks on flat surfaces to represent what they see, or think they see, in the world. All the same, some speculation is justified about possible correspondences between the two uses of the word.

It is important, for example, to note that the lines which the artist draws to represent the outline of an object do not actually correspond to its ~, in the sense that an edge-finding algorithm will replace an abrupt tonal discontinuity with a line. In fact, the edges of an object in the real world are almost never delineated by an unbroken string of abrupt tonal discontinuities. If the artist is unperturbed by the disappearance of the edge, it is likely to be because he i.sn' t using that edge, rather than because he has some efficient algorithn for filling in the gaps. Similarly, most of the objects in the world are occl ooed by other objects, yet it would not normally occur to the artist that the shape of a face is the part left visible by an occluding hand (fig 20).

figure 20.

The face evidently exists for him as a cognitive unit, and will be recorded by means of whatever strategies are appropriate and avail able for the representation (note 15).

It is as true to your meaning of "representation" as to mine, not only that it rests upon the possession of appropriate and avai lable strategies, but also that new strategies may be developed to fit particular concerns. Both are bourrl by entity-spec i fic considerations, however: considerations, that is to say, which are irrlependant of the par ticular event or object being represented and take their form from the underlying structures of the entity -- the artist's cognitive modes on the one harrl and the structural integrity of a computer program on t he other.

Wha t; is ~ Representation "Like"?

It could not be seriously maintained that a computer program is "1 ike" a hunan being in a general sense, arrl it should not be necessary to point out that a representation in my meaning of the ...ord is not "1 ike" the thi ng represented, other than in precisely defined senses of likeness. That may not be quite obvious, however, when we consider the idea that a portrait is "like" the sitter . Even though we may be careful enough to say that the por trait LOOKS like the sitter, or that a musical passage SOUNDS like the rus tl ing of leaves , we tend to stop short of that level of detai l at which it becomes clear that t he a~arance of a pai nted portrait and the appearance of a person actually have very little in conmon. A representation may be about appearance, but we ·never confuse the representation with the real ity, no matter how "lifelike" it is. In fact, we might rat her bel ieve that all representations of a given class are more like each other than any of them is like the thing represented. Life follows its laws, representations follow theirs.

Wha t is ~ Image?

The pur22se of an act of representation i s to draw attention to some particular aspect of the represented object, to differentiate that aspec t from its context, not to reconsti tute the object i tse 1 f. 'Ib that degree we might regard a visual representation as constitut ing a partial theory of that object and its existence, just as we might regard a computer program as constituting a theory of the process it models. But neither the artist nor the program designer has any choice but to proceed

in t erms of the modes which are avai labl e or which they are capable of developing. In the case of the visual representation, the making of an image, I have tried to demonstrate the cogni tive bases of those mcdes, and also, t hrough my own program AARON, to demonstrate their raw power in the image-mediated t ransaction.

That, finally, defines my use of the word "image". An image is a reference to some aspect of the world which contains within its own structure arrl in terms of its-own-5trueture-a reference to the act of co9rlftl0n whlch generated it. It must say, not that the world i s like this, but that it was recognised to have been like this by the image-maker, who l eaves behind this record: not of the world, but of the act.

APPENDIX .!. _ THE WR'l'LE SYSTEM.

When the real turtle is not running, the program simulates its path, and calculates where it would have been in an error-free world after completing each command. In this case it substitutes a chord for the arc which the real turtle would have traced out. (The straight line segments which may just be visible in the illustrations here are due to the fact that they were photographed off the Tektronix 4014 display, not from an actual turtle drawing.)

The Navigation ~stem.

The navigation system is correct to about .2 inches: that is an absolute determined by the sonar operating frequency -- about 40KHz - and does not change with the size of the drawing. Even with so coarse a resolution the feedback operation is efficient enough for the turtle to do everything on the floor that the program can do on the screen: indeed, if the turtle is picked up while it is drawing and put down in the wrong place it is able to find its way back to the right place and facing the correct direction.

The Dynamics of Freehand Drawing.

There are several complexities in this part of the program which are worth mentioning. One of them is that the program has to be able to accomplish dramatic shifts in scale in the drawing, to make sma 11 things which look 1 ike small examples of big things: smoothly-curved closed forms should not turn into polygons as they get smaller. This is required both on the issue of shifts in information density and also to maintain implied semantic relationships between forms.

A second complexity is that the movement of the line should convincingly reflect the dynamics of a freehand drawn line, and this should mean, roughly, that the "speed" of a line should be inversely related to the rate of change of curvature: the pen should be able to move further on a single command if it's path is not curving too radically. (The converse of this is that the amount of information needed to specify an arbitrary line should be a function of its rate of change of direction, with the straight line, specified by its two end points, as the limiting case.)

1-bvement Sea 1 ing.

'111ird, the pen should proceed more "carefully" when it is close to some final, critical position than when it has relatively far to go and plenty of time left to correct for carelessness. This, too, implies a scaling of movement in relation to the state of the local task. Finally, there is the practical problem that for any given nuriber of cycles of a stepping pattern, the actual distance traversed by the pen wi 11 vary with the ratio of the turtle's two wheel speeds. Unfortunately, this relationship is not linear, and neither does it provide a useful simulation of freehand dynamics.

Briefly, the line-generating procedure concludes that, given the present position and direction of travel of the pen in relation to the current signpost and to the final destination, it will be appropriate to drive the two wheels at stepping rates r1 and r2, taking n steps on the faster of~he two.-rn doing so-it takes account of all of the above considerations. The ratio determined for the two speeds is a function of two variables: the angle A between the current direction and the direction to the current signpost, and a scaling factor given by the remaining distance Dd to the final destination as a proportion of the original distance Do (fig 11). This speed ratio then becomes one of the two variables in a function which yields the nunber of steps to be taken-- the distance to be travelled-- by the fast wheel: the other variable being the relative size of the block of space allocated to the current figure.

These functions have to be tuned with some care to be sure that each variable is correctly weighted, and to compensate for the turn­ distance ratio of the turtle geometry itself. But none of this -- or any other part of the program involves any significant mathematical precision. There are only fifteen stepping rates available, symmetrically disposed between fast forward and fast reverse. The whole program, including extensive trigonm~etr ic operations, uses integer arithmetic -- this for historical reasons as well as limitations of available hardware --

APPENDIX II -- Ml\TRIX REPRESENTATION. ---

This description is given here primarily because it offers some insight into the kinds of con­ siderations which the program believes to be important, and the way in which these considera­ tions are accessed: not because there is anything particularly original from a data-structure point of view.

Much of the detail of the implementation is demanded by the word-length of the machine, and would go away in a larger machine. The intent is to make all the information relating to a par­ ticular part of the drawing effectively reside in a particular cell.

The program uses the single words representing matrix cells in different ways according to what is happening in the cells:-

A "simple" event means, essentially, that all the data will be contained within this one word, although it will be seen that its simplicity relates to its use in a more meaningful sense:-

Before beginning work on the drawing, the program "roughens" the surface: that is, it declares some parts to be unuseable for the allocation of space to a new figure, although a developing figure may go into this "rough" space. This is done in order to maximise the rate of change of density across the image:-

and the geometry of the current turtle determines that it can only change direction in increments of about one sixth of a degree. (The turtle was not until recently interrupt-driven, and for design reasons this incremental direction-change factor was one degree in the earlier version.) Everything relies upon the feedback mode of operation to provide correction and to prevent error accumulation. The point is that a good car driver can drive a car with sloppy steering as well as a car with tight steering up to the point where feedback correction cannot be applied fast enough.

I I I I

"Use" may involve either a line or some special spatial designation:=

In either case, the cell will now have a figure identifier associated with it. The new version of the program uses less figures than the earlier one, and develops them further: a maximum of 32 figur~s is permitted:-

If the cell contains a 1 ine, then it can be dealt with as a simple event provided that i t i s not a line junction of a special kind. In this case the entry designates a line function type:-

Cel l Linking.

'!he forward and backward links are a very important device here. Lines are mapped onto the matrix as they are drawn, using an adapted form of Bresham's Algorithm to ensure that strings of cells never include corner-to-corner contiguity. '!his also means that for any given cell, the line it contains must have entered it from, and will subsequently leave it into, only one of four cells: thus the four-bit linking permits a complete traversal of any series of line segments not involving a complex event.

At this point, the single word is inadequate, and it is used as a pointer, words now being allocated in pairs from a freelist. Here again, one level down from the matrix, the words will be variously decoded. In particular, in the event that the cell is occupied by two figures, the two words are each used as pointers to new pairs of words, one for each fig­ ure:-

A cell at this level may contain complex events from one or both of two classes: connective and configurational. Configurational events frequently involve order-2 nodes-- nodes, that is, which fall on a continuous line-- and include sharp an­ gles, strong curvature, and so on. In practise, the program forces complex events so that they always occur within an 8-cell displacement in x and y from another cell, and the location of the next event can then be recorded rather cheaply:-

"Sense", here, means convex or concave if the line is the boundary of a closed fig­ ure, and up/right or down/lef.t if it is not. If this is a figure node of any order other than two, one entry will be needed for each adjacent node:-

in addition to the displacements which chain this node to each of its connected nodes. This means that the traversal of the figure as it is represented by the matrix can con­ tinue from this point until the next node is reached.

Thus, the entire structure is contained essentially within the matrix, and the short lists which may be tacked onto any single cell serve merely to extend the effective capacity of that cell.

Ideally, this matrix should be as fine as possible: since the resolution of high-grade video is only l024xl024, a matrix of this size would obviously constitute an extremely good representation. However, there are two considerations which make so fine a grai n unnecessary. The first is that the program keeps a full list of all the actual coordi­ nate pairs for each figure as it is drawing it, and can access it should some very pre­ cise intersection be required. The second is that the program is designed to simulate freehand drawing, not to do mechanic a 1 drawings, and once a figure is comp 1 eted some approximation to it for purposes of avoidance or even intersection is unobjectionable. The maximum error induced by assuming a point to be at the center of a cell in a matrix of 90xl60 will be about 7/Sth of an inch in a sixteen-foot drawing: only three times the thickness of the line.

note 1. The word "representation" is used here i n a more general sense than it now carries wi thin the A.I. community: the problem of formulating an internal (machine) representation of some set of knowledge differs from the more general problem primarily in its t echnological aspects.

note 2. "The Art of Artificial Teams and Case Studies Engineering," Ed Feigenbaum, IJCAI5, 1977; pp.1014-1029. Intelligence: 1, of Knowledge Proceedings of

note J. In the decade before I became involved i n my present concerns my work was exhibited at all of the most serious international shows, ~nd I represented my country at many of them, 1ncluding the Venice Biennale: as well as in some fifty one-man shows in London, New York and other major cities.

note 4. Different from each other, loosely speaking, in the way one might expect a human ar t ist's drawings to differ one from another over a short period of time.

note 5. Written in "C", under operating sustem. the UNIX

note 6. I am referring here to differentiations per formed in relation to the image, not in relation to the real world, with which the program has had no visual contact.

no te 7. The program does not attach semantic descriptors to the things it draws: the terms "penumbra", "boulder" and so on are my own descriptions, and are used here for the sake of simplicity.

note 8. Significantly, from the point of view of my argument here, the dirty marks were interded to "suggest" the elements of a composition.

note 9. The one unconstrained randanising agent in this scenario, the final cutting of the film by the producer rather than the director, has also demonstrated itself to be devastatingly non-creative.

note 10. "Undenied" is stressed here because there exists an odd case in which the will of the artist is to produce objects which demand the contemplation of their own qualities for their own sake -- what they are rather than what they stand for -- and which thus seek to deny the viewer his normal assumptions. TO the degree that this aim can actually be achieved the resulting object could not properly be called an image, and I doubt whether aesthetic contemplation could properly be called reading. Thus much of XXth Century abstract art falls outside this discussion.

note 11. It is worth noting, though, that AARON did mechanical straight-line shading for about two years -- it ran faster that way -- and in that time only two people ever remarked on the inconsistency.

note 12. I will leave aside the interesting question of whether there are not more general underlying structures which are coomon to all physical experience. It is presumably no accident that terms like "repetition", "closure", and others I have used in relation to visual cognition are freely used in relation to music, for example.

note 13. The control of the rate of change of information density across the surface of the image, to which I referred earlier, is the most powerful example I know in this regard. The eye is capable of handling units as small as a speck of dust and as large as the sky, but the processes which drive the eye seem always to adjust sane threshold to yield a preferred distribution spanning only a few octaves.

note 14. In fact, the more theatrical explanations which range from world-wide migrations to the influence of extra­ terrestrial voyagers are not even necessary.

note 15. He is unlikely to treat the boundary between face and hand as part of the face, but as part of the hand, and may very well indicate the full boundary of the face as if he could actually see it.