A Framework for Handling Vision Data in an Object Level Robot Language - RAPT

1. Introduction

There have been up to now two major approaches available for programming robots [7]. One is teach-node programming: this is well known as pro­ g ramming by showing. The other is off-line pro­ g ramming and is referred to as programming by us­ ing some formal language. Since the off-line pro­ g ramming method has many advantages over the teach-mode one, there is a trend for ·robot users to program industrial robots by using robot languages.

It is also believed that if sensory informa­ tion is used in conjunction with the robot, then the robot can do a better job . In this paper we will discuss the work of combining a special kind of vision, verification vision, with a particular robot language- RAPT [1,2,3,4]. However, the au­ thor believes that the principle of the work can also be used to deal with some other kinds of sen­ sory data.

2 . The Current RAPT Language and Its New Vision Col'\mands

2.1. The Current RAPT Language

RAPT is a model-based object level robot com­ mand language [6,7]. Object level languages allow the human user to describe the task that he wants the robot to perform by describing the objects that are to be handled and the things he wants done with them in terms that are natural to him rather than to the robot. This information has to be converted by some computational system into run tir.te commands that the robot can obey. In RAPT, the environment of the robot is modelled by an in­

complete geometrical modelling system. Each ob-

* This research is supported by a Chinese govern­

ment studentship. It is also supported by the University of Edinburgh .

A FRAl!EI/ORK FOR ll!IADLillG VISION DATA Ill AN OBJECT LEVEL

ROBOT LANGUAGE - RAPT

Yitl Baolin

Department of Artificial Intelligence

Un iversity of Edinburgh , Scotland

ject in the environment is represented by those of its features which are to be used in the associat­ ed P~T program . These features, such as planes, edges, points, etc ., may be finite or infinite . In P~T programs, the objects which are to be manipulated by the robot a re explicitly represent­ ed, and the programmer specifies intermediate states a nd actions on objects. These are transformed by the RAPT system into movements of the robot which will bring about the desired goal

state.

RAPT is designed mainly for doing automatic assembly a nd allows the user to specify a set of bodies, the spatial relationships that are to hold between their features in each goal state, and their movements between one situation and the next. Spatial relationships constrain the rela­ tive positions of bodies. If multiple spatial re­ lationships hold between two bodies then new, r.wre restrictive relationships may be deduced to hold between the bodies. This ability is a result of spatial relationship reasoning.

The output of normal RAPT is a series of posi­ tions of objects in each situation. These are only planned positions in as much as they have been determined taking no account of the inaccura­ cies inherent in the real world . The purpose of the current work is to allow statements to be ad­ ded to the RAPT system which will allow use to be made of vision data to modify the planned posi­

tions.

The work of the RAPT system is divided into two parts: compile tine reasoning and run tine ex­ ecution . At compile time, the system performs off-line reasoning about the positions of the ob­ jects and the actions of the robot. The results of the compile time reasoning will guide the ac­

tions of the robot at run time.

Readers who are not familiar with recommended to refer to [1,2,3,4] understand the paper better. RAPT are in order to

2 .2. Verification Vision and Vision Task Specifi­ cation in RAPT

A wide assortment of devices and systems exist for robot sensing of the environment. They fall into two generic classes: contact and non-contact [8]. Vision is one of the most important non­ contact sensing methods. Verification vision, as a type of visual information processing, is quite different from other types of vision is very useful in conjunction with nain characteristics that distinguish systen, and robots. The it from others are [5,10]:

1. the systen has a great deal of prior knowledge about the type, placement and ap­ pearance of the objects that form the scene,
2. the goal is to verify and refine the loca­ tion of one or more objects in the scene
3. rather than to recognize then.

Defore the verification system can be used, it nust be told exactly what to look for and approxi­ nately where to look. The error between the ex­ pected ("nominal") positions and the actual ones

should not be too big .

Since RAPT reasons about spatial relationships between body features, we nay ask the vision sys­ ten to verify the positions of sone special features, e . g . edges, and then send the vision data to the l~T reasoning sys tem so that the ac­ tual position of the body to be verified can be deduced. Since inages of edges are easy to detect

\Je prefer .to use edges in the sys ten.

To introduce verification vision facilities into the RAPT systel:l, a nunber of vision commands nust be added to the RAPT language . They are the LOOK s tatenent, the IllVIOLATE s tatenent, the TOLERANCE statement and the CO!!BINE statement. Sone auxiliary comnands are also needed in order to specify cameras and some low level vision de­ tails .

The LOOK statement is used to specify the features to be verified by the vision system . Its main effect is to form symbolic features and rela­ tionships in the RAPT reasoning network. It also sends the necessary information to the low level vision systen to enable ·it to find the expected edge image at run tine.

\/hen a camera has been used to find the inage of an object feature in the scene, the RAPT system creates a ne\~ virtual feature for the camera and establishes a spatial relationship between this new feature and the observed feature of the ob­ ject . For example, it may establish an "against" be tween a newly created plane feature of the cam­ era and an edge of the object. Since the relevant vision data is not available at compile tine, the spatial relationship, will have symbolic forms

new featu're of the camera, and therefore the new during the conpilation phase.

The INVIOLATE statement specifies a constraint on the position of the object to be verified in terns of a relationship that nust hold between the object and the world. For example, INVIOLATE can be used to indicate that the bottom of the object nust be against the top of the table no matter what the inac·curacies of the placing of the object are. From the view point of geometric reasoning, it provides a reliable relationship in the rela­ tionship network. This will enable the geometric reasoning system to explain vision data in a

correct way.

':'he TOLERAllCE statement specifies the maximum translational error along all the three axes of the body coordinate system. The rotation error restriction of the translation error should indi­

tolerance is not discussed in this paper. The cate the range in which the feature is likely to be found.

':he C0!1RINE statement provides a package for the vision task. It invokes the symbolic reason­ ing facility to deduce the symbolic position of the object by use of all the information included in the package . It checks whether the statements in the package are compatible or not. It also combines the information given by the IllVIOLATE and TOLERANCE statements in order to deduce real restrictions on the position of the object to be

verified.

The fornat of the package of the vision com­ mands is like this:

CO!!BillE; VIOLATE/ against, bottom of bodyl, top of table; LOOK/ edgel of bodyl, cameral; LOOK/ edge2 of bodyl, canera2; TERCOI'1;

where TERCO!! terminates the COI1BINE package . De­ tailed discussion of these vision statements can be found in [24].

2.3 . The Symbolic Reasoning Facility

The symbolic reasoning facility is required in the RAPT system for dealing with vision data at compile time. During compile time, the positions of features which are created by LOOK statements have only a symbolic forn and cannot be evaluated until run time, when the vision data is acquired . In this case, the RAPT inference systen must deal \~i th the symbolic form of the feature positions

rather than their actual values . The result of the process will be evaluated during run time when the real feature positions have taken the places of the symbolic ones. Therefore, a symbolic rea­ soning facility is essential for combining sensory data with the RAPT systen.

The symbolic reasoning systen works in a simi­ lar way to the current RAPT cycle finder [14]. Being given two relationship chains between two objects, it will produce a constrained new rela­ tionship between the objects by means of a set of rewrite rules. The difference between the symbol­ ic reasoning system and the cycle finder is that the features in both the input and output of the symbolic reasoning system may be symbolic and

their parameters may be symbolic expressions. De­ tailed discussion of the symbolic reasoning facil­ ity can be found in [24].

3. The Franework for Handling Vision Infornation

So far we have discussed how to specify a vi­ sion task and how to reason about vision informa­ tion symbolically. Now we will discuss how to provide a framework to handle the symbolic position expressions caused by introducing verifica­ tion vision.

3.1. Outline of the Framework

It is commonly the case that v~s~on data \lill verify not only the position of the specified body at the current situation (body instance), but also some other body instances whose positions are relevant to or deduced from- this body instance. The verification v~s~on indicates discrepancies bet11een an expected position and an actual posi­ tion in a particular situation and it will be necessary to modify subsequent positions in the light of this information. For example, suppose a robot moved a block to a specified position and then moved away a fixed amount waiting for the vi­ sion system to operate. If the verified position shows that the block is not exactly at the speci­ fied position, then we are sure that the robot hand is also not at the position where we supposed it to have been.

He \vant to avoid, as much as possible, reason­ ing which involves the symbolic expressions representing verified positions. Otherwise these expressions, or parts of then, would appear in a large number of places in the run tine code, and the evaluation of each of them would take too much time. On the other hand, if we can find a method of using the verified position only at run time, then the run tine system will work faster. Here we outline a method of using verified positions at run time only. We will employ two reasonable as­ sumptions in the following discussion. They are:

1. The nominal position of a body is assumed to be accurate unless there is some evidence
2. (e.g. vision data) to the contrary.
3. The movement of _a robot arm is assumed to be accurate over small distances.

3 .1.1. Analysis of Expressions of the Body In­ stance Position

\le first need to establish how the actual po­ sition of a body instance is related to its nomi­ nal position and the vision verification data, even after some movements have been made . A move­ ment of a body (b) in one situation (i) to the next (i+1) can be represented by a matrix Tb(i+1) such that

where PNbi and PNb(i+1) are matrices representing the --nominal positions of the body in situations i and i+1. If we consider that the body b makes a virtual movement from its nominal to its virtual position then the movement can be represented by a matrix FHbi:

and we call Flllii the modifying factor of the body instance PNbi.

\le know from (2) that:

!low suppose that the nominal position of the body bin situation j, PNbj, is produced by a se­ quence of specified r.10vements from its nominal po­ sition in situation i, then it can be seen that

where

The actual position of the body in situation ; (the actual position is not a "verified position;' since we have not verified it by vision commands, but \le consider that it is equivalent to a veri­ fied position in our discussion) will be related to the actual position of the body in situation i by exactly the same sequence of movements, and therefore

Flfuj is referred to as the modifying factor of the body instance PNbj.

\le can see from the discussion above that the llhen a body in­

actual position of a body instance is determined by two parts: the nominal position and a modifying factor, and it is only the modifying factor that is affected by the vision data. stance is verified, its modifying factor is de­ fined by the vision data and by the nominal posi­ tion of the body instance. Furthermore, if we know the modifying factor of a body instance and the subsequent nominal movements of that body, then we can determine the modifying factors for the instances of that body in the subsequent si­ tuations. As we can obtain the nominal position for each body instance by using the current RAPT cycle finder system, we can determine the nominal movement of a body instance between any situa­ tions ·. Therefore, we can obtain the modifying factor for every body instance by working forward from the modifying factor of the verified body in­ stance. We can see from equations (6) and (7) that we can also assume that once the actual posi­ tion of a body has been found to be different from its nominal one, accurate positions of the body in subsequent situations will bear the same relation­ ship to their nominal positions until a new modi­ fying factor is found either by new vision data or by a specified action (see Sec. 3.2.1). There­ fore, in our vision verification system we may deduce the nominal position of each body instance at compile time in the usual way, then evaluate the modifying factors at run time, and get the ac­ tual positions by matrix multiplication.

The introduction of modifying factors will af­ fect the actions of bodies. We will refer to the action bet~een t~o nominal positions as nominal action and that bet\~een tHO actual positions as actual action . Suppose body b is moved from nomi­ nal position PNbi to PNb(i+1) by a nominal action Tllbi. lie have

On introducing the modifying factor, the posi­ tions of both the starting point and the destina­ tion of an action may be changed. Suppose body b is moved from actual position PVbi to PVb(i+1) by an actual action TAbi, then we have

PVbi * TAbi = PVb(i+1) Fl!b i i: PNbi * TAbi = Ft!b(i+1) * PNb(i+1) TAbi PNbi- 1 * Fl!bi-1 * H!b( i+ 1) * PNb( i+ 1) (9)

3 .1. 2 The Run Tiue Data Structure

The modifying factors are stored at run time in an array indexed by body instances. The ele­ ments of the array may have three forms . The modifying factor will be an identity matrix symbol ("I") if the corresponding body instance can be assumed ·to be in its nominal position. The modi­ fying factor will be a position matrix .if the corresponding body instance is verified by vision commands . The modifying factor will be a pointer if the position of the corresponding body instance is dependent on another body instance which has been verified in the current situation or one of the preceding situations. The pointers and iden­ tity matrix symbols are assigned at compile time, while the position matrices are assigned at run time after the vision data .have been obtained and the corresponding symbolic positions have been evaluated .

There are two possible ways to use the modify­ ing factor array in controlling the actual action of a robot. In the first method we use equation (9) to calculate the actual action directly and then order the robot to move accordingly in one step . In the second method we divide the actual action into two parts and control the robot to move accordingly in two steps . The first part in­ dica tes the action which is caused by errors in the end points of the motion while the second part indicates the nominal movement specified by the RAPT program. From equation (9) we have

= TCbi * Tllbi where (10) TCbi = PNbi-1* FMbi-1* Flfb(i+1) * PNbi (11)

The robot moves by an amount of TCbi in the first step and then moves exactly the amoun t indicated by the RAPT program to achieve its actual destina­ tion. The second method is important in some cir­ cumstances and is discussed in detail in [24].

3.1.3. Compile Time ~lark

The work to create the run time modifying factor array can be divided into t~o s tages, the rea­ soning stage and the simplifying stage. In the r .easoning stage, all the modifying factors for the body instances in the initial situation will be assigned an identity matrix symbol ("I"). In each of the following situations, if a body is verified in that situation the corresponding modifying fac­ tor will be assigned the symbolic position expres­ sion PVbi ("P"), otherwise a pointer' or a set of pointers which points to another modifying factor will be assigned to the modifying facto r according to rules which will be discussed l ater .

In the simplifying stage , the modifying factor array is sir.1plified according to a set of rules. This process makes the array more compact and speeds up the run time evaluation of the array.

3.1.4. l!odifying Factors in Symbolic Reasoning

As well as influencing the symbolic reasoning, the introduction of nodifying factors will also influence the way vision commands are used. The modifying factors from previous vision commands may enable us to make better predictions for posi­ tions of bodies in subsequent vision commands.

In a vision command package , some INVIOLATE statements may be used to indicate the most reli­ a ble relationships holding between the object to be verified and some other objects. The modifying factors for the body instances of .these reference objects may not be identity matrices. This means that they may not be at their nominal positions, though the relationships mentioned in the IN­ VIOLATE statements s till hold. Discussion of how to deal with these two points can be found in [ 24] .

3.2. Rules for Setting the Pointers

The rules discussed below are used for setting the pointers in the modifying factor array for bo­ dies according to their status in the RAPT program such as being MOVEd, TIEd and so on. For conveni­ ence, ~e ~ill refer to the rules for making the pointers as linking rules.

3.2.1. Actions

':!:'here are two kinds of action statements in ru\PT: I!OVE statements and TURN statements. For making a modifying factor array, however, we are more concerned with spatial relationship specifi­ cations about bodies and the association between the relationship specifications and action state­ ments. If the position of a body instance is res­ tricted by some spatial relationship specifica­ tions, then we refer to the position of the body instance as a specified position. On the other hand, if there are no explicit spatial relation­ ship specifications constraining the position of the body instance, then we refer to the position as an unspecified position.

7he following are the linking rules for the body instances which are moved (i.e. MOVEd or TURild) directly by action statements rather than by the effects of TIEs and SUBASSENBLIES. The linking rules for ':'IEs and SUBASSE!!BLIES will be discussed in Sec. 3.2.2 and Sec. 3.2.3.

• Al. If a body is moved to an unspecified posi­ tion, then the pointer of the body instance points to its r.10difying factor in the previ­ ous situation.
• A2. If a body is moved to a specified position, then the pointer of the body instance points to the modifying factor for the body instance of the body be·tween which the relationships hold (reference body) in the current situa­ tion unless the -condition mentioned in rule (A3) is met.
• A3. If the reference body in a specified posi­ tion is TIEd to the body to be moved or be­ longs to the sane subassembly as the body to be moved then the pointer of the body in­ stance points to its modifying factor in the previous situation.

3 . 2.2 . Ties

In the RAPT language bodies can be tied to­ gether during an action, and this means that they maintain the sa~e relative position before and after the action. Therefore, any descriptions of the motion of one body which is tied to another must apply to the notions of that other. ~IEs are

made and revoked by TIED and UNTIED statements.

We can see, from the definition of TIE, that the modifying factors for bodies which have been tied together must keep the same relations to each other throughout the existence of the tie, except when .local vision commands (see [24] for details) are used. This means that a change in the modify­ ing factor of one member of a tie must be applied to that of the other member. lloreover, this ef­ fect will continue after the t\>TO bodies are untied until one of them is moved by specified actions or both of them are verified by global v1s1on com­

mands individually (see [24] for details).

Let us consider an example in order to under­ stand how to keep the same relationship between two modifying factors when one of them is chang­ ing. Suppose bodies A and B are tied together with different modifying factors Mal and ~ful respectively. If l!al is changed to Ha2 then the relative change for modifying factor Mal is Tr. He have

and the new modifying factor l1b2 for B is

Therefore the modifying factor for body A in the new situation is !1a2 while Hb2 can be represented by a pointer triple [pl,p2,p3] in which pl points to lful, p2 points to Hal and p3 points to Ha2. At run time the triple will be evaluated by the equa­ tion

':'he discussion above can be expanded to cover a tie linking more than two bodies and the cir­ cumstances when this large tie has been broken. Suppose n bodies bl, .·· , bn are tied together by n-1 TIE statements. For convenience we will call the result a "super" tie in the following discus­ sion.

The linking rules for TIE statements can be summarized as follows:

• Tl. In a super tie, if body bj is moved in si­ then the pointer of body bj will point to the modifying factor for body C in other bodies in the super tie are pointer
• tuation i by a specified action which brings about some relationships between body bj and body C, situation i \lhile the modifying factors of triples. The triples have the form [pl,p2,p3] where pl points to the modifying factor for the same body in situation (i-1), p2 to the modifying factor for bj in situa­ tion (i-1) and p3 points to the modifying factor for bj in situation i.
• T2 . In a super tie, if body bj is verified by a set of global vision commands in situation i, then its modifying factor will be assigned a symbolic position expression "P" while that triples. The triples have the same form and
• of other bodies in the super tie are pointer contents as in rule (Tl).
• T3 . In a super tie, if body bj is verified by a set of local vision commands in situation i, then its modifying factor will be assigned a symbolic position expression "P" while that of other bodies in the super tie \vill refer to their modifying factor in situation (i-1).

Five similar rules are applied after a super tie has been broken. Details can be found in [24].

3 . 2.3. Subassembly

The subassembly is a set of bodies between whose features certain specified relationships hold for the duration of the existence of the subassembly. Subassemblies differ from ties in that there may be more than two bodies within a subassembly and the components of a s-ubassembl y can move with respect to each other during the ex­ istence of the subassembly, provided that the re­ lations remain valid. Treatment of subassemblies is similar to that for TIEs. Detailed discussion can be found in [24].

The following simple example shows how the linking rules are used.

remark bodies bl b2 b3 have been defined; verify/b2; remark abbreviation for vision command package, now in situation i;

move/bl;

fixed/bl, b2; remark abbreviation for a set of relations which completely defines the position of bl with b2, sit i+l;

4. The Position of the Camera

The camera is defined in RAPT in the usual tray as an ordinary body with a specified focal length. This means that as with any other body it can be operated on by the system, being moved, tied and so on. It can also be mounted on a robot hand. In most cases the position of a camera body can be modified by the modifying factor array according to the linking rules . For example, the position of one camera can be verified by another camera . However, if the result of a g lobal vision command package will affect the position of the camera to be used (according to the linking rules listed in Sec. 3.2) then the vision command package will be dealt with as a local one. In this case, only the position of the body to be verified .is subject to modification while the position of the camera is unchanged.

5 . The Generality of the FranetJork

So far we have discussed how to use the frame­ ~mrk to handle verification vision infomation. In fact, the framework also has the capability to handle other kinds of sensor infomation for the purposes of verification. Generally speaking, in RAPT, Hhen we use a sensor to detect a feature of an object, we create a new relationship between that object and the world, as the absolute posi­ tion of the sensor in the world is usually known when the sensor is used. The nature of the rela­ tionship depends entirely on the type of sensor. Since sensor data is available at run time, at compile tine we can only obtain symbolic relation­ ships. \lhen the s·ymbolic relations are sufficient to "fix " the object, we can use a suitable symbol­ ic reasoning system to deduce the synbolic posi­ tion of the object. We can then evaluate the sym­ bolic position expression at run time when the corresponding sensor data is available . In order to introduce a new kind of sensor, we need only to provide a set of commands to specify how to use the new sensor, and a new symbolic reasoning sys­ tem which is capable of dealing with the new rela­ tionships created by this sensor. The framework handles new kinds of sensor information in the sane way as it does vision data.

Now let us consider an example . Suppose we use a touch sensor to detect a plane feature pl of a body B. When we detect a point on pl, we create a relationship between the ~mrld and the body, the "AGAUIST" relationship bet1<1een a sphere feature (a vertex is considered as a sphere with radius 0 in RAPT) of the world and the plane pl. If we detect some carefully chosen planes of the body B with the touch sensor and so obtain enough AGAINST re­ lationships, then \Je will be able to deduce the position of the body B. The symbolic position ex­ pression of B can be stored in the modifying fac­ tor array and can be used in exactly the same way as verification vision data.

Some authors (e.g. [15]) have suggested that vision sensors have poor precision and therefore it is better to use both vision and contact sen­ sors. Vision sensors can be used for coarse sens­ ing while contact sensors can be used for fine resolution. In this case , the generality of the framework to handle both kinds of sensor informa­ tion is very important.

6. Discussion

There are two main problems in using sensory infomation in programr.ling and controlling robots. The first is how to get accurate sensory informa­ tion and the second is how to process and use the information in the system. The latter problem, as Hamon points out [8], seems the harder of the two and has more implications for the theoretical use of sensors.

Some work on using vision data in robot pro­ gramming languages has been reported, such as VAL [19, 23] and AL [20]. In the VAL system, v1s1on is used to recognize objects in a picture taken by a TV camera, and to locate them. The recognition is done by comparing the blobs in the picture with trained characteristics of given prototypes in an effort to find a match . The vision technique used in the VAL system is of the first generation [12], i.e. it uses a binary picture and produces a 2-D explanation. In the AL system, vision is used to fulfill the verification task [20]. Both the languages mentioned above are classified as end­ effector level languages by Koutsou [7], i.e. only the end-effector of the robot appears explicitly in the program.

Some object level languages have the ability to use some touch sensor information . For exam­ ple, in AUTOPASS [21] force or torque sensor in­ formation is used to provide special threshold values in order to constrain or terminate some ac­ tions. In the LA!~ system [22], the force sensor is used to detect whether a specified action ter­ minates at a correct position or not. In both the above systems, sensory information is used as a condition in a decision tree and no further expla­ nation of the information is made.

In contrast to these systems, the RAPT language with the verification vision facility provides a method of specifying vision tasks and reasoning about the vision data symbolically off­ line in an object level language. It makes no as­ sumptions about the relative positions of objects and sensors as a teach-method does. It allows partial information about positions to be combined with sensor information in a general way. At run time, when the vision data is available, the results of symbolic reasoning are evaluated and are used to improve the system's knowledge about its environment. This method uses grey level pic­ tures and produces a 3-D interpretation rather than a 2-D one. The framework described in this paper enables us to combine the verification vi­ sion facility with an object level language in an intelligent way. The framework which is used by the system for handling vision data has some gen­ erality. It is not only able to handle the vision data, but also able to deal \>'ith sor.1e other kinds of sensory infornation, such as touch sensor in­ forr.Jation. Using the framework to ir.1plement the vision facility within RAPT requires few changes in the current RAPT system.

In this paper we have only discussed the theoretical framework. The system has been imple­ r.Jented within RAPT and used with sir.1ulated vision data, but as yet no link has been made with a real vision system. This will, of course, provide a real test, and we anticipate some problems in en­ suring the correctness of match between the r.1odel features and their images.

Acknowledgenent

I ar.1 grateful to A. P. Ambler and R. J. Pop­ plestone for their supervision. I would also like to thank P. Brna, D. Corner, S. Cameron, R. Feath­ erstone, R. Fisher and J. Hallam, for their help­ ful criticisms of my work.