Visual-Tactile Symbiotic System for Stereometric Pattern Recognition

Introduction

Solving problem of stereometric pattern recognition is important fqr realizing practical artificial intelligence systems. It appears likely that dynamic visual-tactile information processing will be used in several such man­ machine systems. In this paper we will describe mainly the tactile information processing techniques for designing a symbiotic system.

Some studies of tactile displays have been made; for example, a conference on tactile dis­ plays was held at Stanford Research Institute, Menlo Park, California, on April 3 and 4,1969. A major objective of this conference was to examine current neurophysiological and psycho­ logical theories to determine the extent that a rational basis for the design of dynamic tac­ tile displays could be developed, (1)

VISUAL-TACTILE SYMBIOTIC SYSTEM FOR STEREOMETRIC

PATTERN RECOGNITION

S. Aida, L. Cardella and N. Ivacevic

Laboratorio di Cibernetica del CNR Arco Felice, Napoli, Italy

In general, the tactile model includes a pr·ssure-electric transformation element, and this system is modeled after its physiological counterpart.

It is well known that the sensory nerve and its ending which controls the tactile sense exist in skin. The particular model eliminates the function of the temperature, pain and touch sensation of the tactile sense and arranges a piezo-electric element of strain sensor corre­ sponding to pressure sense.

The tactile sensor to be described here consists of normal on-off elements. The fea­ ture of this tactile system is to provide the finger with tactile sensors to move along the surface of a object when one of the sensors

touches it.

Stereometric pattern recognition is made use of in the system for manipulating objects. The particular device for manipulation is called a manipulator; it is an artificial hand with a tactile sense. We consider these sys­ tems to correspond to human sense organs and call them visual and tactile systems, respec­ tively. The visual system is responsible for spatial information and the tactile system for detailed shape information of objects. The purpose of this symbiotic system is to provide a kind of artificial intelligence by using both visual and tactile information.

System design is an important part of this kind of study. In order to provide a framewo-rk for this symbiotic system, a general approach to its system design has proposed by Aida and Kinoshita (2), the experimental structure is

illustrated there.

System Design

An important feature of this symbiotic system is that the information processing is carried out by coordinating visual and tactile information. The system described here can be actualized in two forms: one is an open-loop structure and the other a closed-loop one which

includes learning techniques.

In the open-loop symbiotic system, the output of the visual sensor serves as an input to the tactile sensor, and this system can rep­ resent a time sequence of events from visual to tactile sensor. A simple block diagram is il~ lustrated in Fig . 1 (a).

Although in Fig. 1 (b) the visual sensor and the tactile sensor are in chronological sequence, the results from the tactile sensor are fed back to influence what will later occur at the visual sensor, for example, a grey level or scanning ~e~hod control. The resultant out­ put of the ~ombination appears as the ~age of a visual display, and the effects of changes in the individual b~ocks o., the resultant output are different for the open-loop and closed-loop con­ figurations.

In the open-loop symbiotic system, visua: processing is only used to decide the position of the object, whose image is obscure, then the tactile sense moves towards the object and touches its predetermined part to get a detec­ tion matrix for the object. After matrix pro­ cessing, tactile information is converted . into a visual signal to present a clear image of the visual desplay, but this process has only been for the edges of the object.

Therefore, pattern recognition by the open loop symbiotic systems is carried out step by step as follows:

1. Range finding and locating of the di­ rection of gaze with respect to visual sensor. Visual image of the object is partly or completely obsucre.
2. Positioning of obscure part by the di­ vided area on the image screen.
3. Tactile seasor moves to the object and touches the obscure part of the object on the images screen .
4. Detection matrix is made on the basis of 3).
5. Convert tactile information to visual display signal.
6. Loop 3) to 5) until the obscure part on the image screen is cleared.

A block diagram for the open-loop and the closed-loop symbiotic system is shown in Fig.2, it shows conceptually a block diagram of the symbiotic system with visual and tactile senses. This system consists of four parts; a part which processes information from the visual sense, for example, a vidicon camera, a part which processes those from the tactile sense, a part which inte­ grates the information, and a display device which indicates the process.

The ir>.itial scene of objects obtained by the vidicon camera and its displayed image is ob­ scure. It has been impossible to recognize ob­ jects with this primary image. only. However this initial information establishes the existence and the position of the objects. Variations in brightness are extracted by using the procedure of lateral inhibition, and then the digitalized image is cut off by an appropriate threshold val­ ue. The process provides the beginnings of fea­ ture extraction and the recognition of objects, but for indestinct features. The finger with the tactile sensor is stretched out, and the features of the parts of objects are .extracted by being touched with the finger.

The tactile system searches out and recog­ nizes states of the object surface for obscure parts of the picture. The feature is taken by using a contact distribution which takes account both of tactile information and of the finger's movement . The feature on the image is transformed to a line drawing picture which is processed with the detection matrix, elements of which are abstracted by the contact distri­ bution generated by the tactile sensor. The function D (u,v) of two functions p(u,v) and A (u,v) ~~ shown in Fig.3 is defined as mn

It is considered that the output D indicates the same features pattern in the wiWSow A as the pattern of the detection matarix. IfmPhe output equals zero, the feature of object is not extracted in the window.

Then, the system adjusts its parameters, which are gradually varied with the observation of D in the window established from the result of r:2ognition. Three kinds of modified parame­ ters are considered, they are processed acording to the following order: 1) focus of vidicon camera, 2) contrast, 3) elements of detection martrix. First, it is necessary to adjust the focus so that the clearest feature of the pat­ tern can be obtained. If the focus is off the point, the image becomes obscure. In the sym­ biotic system, the focus is adjusted by a visu­ al sensor driving signal as shown in Fig.2, and the distance from the position of finger is calculated.

Second, the constrast is gradually adjust· ed. If the contrast is highly intensive, the feature of pattern is lost. The Amn in the picture is established as the unknown part for the recognition of the picture and stored pat­ tern. As the part of the object corresponding to Amn is recognized by the tactile sense . parameters ~f visual system like contrast or brightness are adjusted ao that the recognition will coincide with that from tactile sense. Moreover, a shadow pattern of the object is varied with chages in the derection of the light thrown on it and the feature of the pattern is extracted more easily.

Third, the element of detection matrit is changed correspoding to the sensitivity valu~ of D . As described above, the image obtained frommrh~ visual scene is transformed automati­ cally into a scene with clear edges . With this image, the whole scene is recognized.

Thus, in the closed-loop symbiotic system, whenever the·object is touched by the tactile sensor, the tactile information controls param­ eters of visual sensor, such as the thresho~d level of the visual-processing system, and a clearer image results.

Artificial Tactile Perceptions

The assumption underlying tactile explora­ tion is that if a device i.s able- to feel a con­ tact situation with the object and to identify automatically the surfaces which limit i~, then it is possible to get a quantity of information sufficient to enble a computer to characterize :he object from a geometrical point of view.

Using sensors of a particular kind, it may be possible to characterize an object also from other points of view, thermal, chemical, and so on.

The computer can make some elementary deci­ sions about the shape and the kind of the exam­ ined object and subseq~ently can give some simple orders to an actuator.

Computer Simulations of Stereometric Forms

A lot of methods have been proposed for visual processing (3). This section, therefore, describes some computer simulations of the tac­ tile-processing system which is under develop­ ment to realize this kind of system. It is difficult to say how large a part the tactile sense plays in the field of pattern recognitions. As the first step in this determination, we con­ sider the exploration of space patterns by di­ rect contact only, that is, by the sense of touch. Then, we are trying to optimize the prehension of objects at a lower level in the hierarchy of manipulation control, dependent on their stereemet·ric forms.

In order to ac=omplish a classification of stereometic forms we used the propositional cal­ culus with operations already known from Boolean algebra. Each proposition is an affirmation or negation of the presence of a certaion feature which an object could possess. A certain com­ bination of propositions gives a mathematical model of a spacial form. This procedure make it possible to use the cam~~ter in the process of the classification and recogniLion of spacial patterns. If the concept of the object corre­ sponds to that of the abstract model. The answer should be satisfactory.

Obviously, it is not necessary to take into account all established propositions, bUL only characteristic ones, because complete classifi­ cation hopeless if all possible inputs are considered. However, classification carried out in phases- A, B are classified in phase I, and C, D in phase II- and manifests a large economy as regards the complete classificatory system.

As an example, a choice tree is gt¥en, as shown in Fig. 4; it consists of a series of phases by means of which one gradually arrives at more precise identification.

In the decision process, we sta~:t with the assembly of initial propositions, which form the basis of the calculus, marked with:

From n elements of the base L, we can theoreti­ cally form 2n members of all possible combina­ tions as follows:

(The sign over the letter means negation)

If some basic propositions are joined by logical connections, the disjunction s(L) will have some empty members (the values of which are zero). In this c~se, the number of conjunc­ tions is reduced to 2 - k, where k is the number of eliminated conjunctions. As k becomes bigger, the logical connection between basic propositions becomes stronger. The re­ maining conjunctions form the reduced disjunc­ tion r(L) which characterizes one determined object, i.e., a mathematical model which can easily be represented by a string of numbers.

Logical connections between the proposi­ tions Li could be carried out in different ways. Here we have used the form of equality, which, within certain limits, can be treated as an equation and processed numerically on the com­ puter.

As an example, here is a group of chiracer­ istic propositions with the marked phases of classification mentioned before:

I A There is contact with the object.

II B Surface is curved.

C Surface is plane.

III D "Vertical" edges are parallel.

E "Vertical" edges connect at one point.

IV F Three pairs of parallel surfaces exist.

V G Distances between neighbouring vertices are equal.

.......... etc.

This group is not definite, it can be changed and enlarged, depending on the principal task of classification.

Combinations of propositions existiogtin the description of the object do not exhaust all theoretical possibilities. Besides conclusions of bogical connections between propositions which allow us to distinguish between objects, it is also possible to find common conditions for tae variants in one class of objects.

Considering the disjunction chracteristics described above, the logical connection for a cube, using the equality form, would be:

giving, as a result, a new proposition:

or Because the density of tbe sensors is low the model obtained is more or less approximate: In the beginning, we limiLed our exploration to simple objects with plane surfaces.

Input date have to be obtained by scanning the object. Any touch of the object by the sensors corresponds to one plane; the equaLion of this is easy to find using analytic geometry. In this way, the touched body will be repre­ sented by one polyhedrDn which envelops the body and waich is coaposed of the group of planes whose equations are known. Co-ordinates of the vertices and equations of the edges are easy to calculate. (The equ·ation of the support plane is the same as the equation of the sup­ ported surface of the polyhedrDn).

If we suppose that an object is touched at point p (determined by the vector rl) by_the sensors (determined by the unit vector r ), then the equation of the tangential plane 0 is:

Then, if the object is scanned by touching in more points, we can have the assembly of all equations of the surfaces of the enveloped polyhedron, which gives us the mathematical model of tbe object.

A flow chart of stereDmetiic forms· using tactile information is showg in Fig. 5. In the flow chart an input matrix is obtained by tac­ tile exploration of the object, meas~ring the co-ordinates of the touch points and the two angles which determine the tange.Dtial plane in each point, using a convenient reference system. The first three numbers in one row are Cartesian co-ordinates : x, y, and z respectively, and the following two are values of the azimuth and inclination angles. In this way the matrix has five columns and as many rows as there are touch points with different tangential planes.

In the present stage of the work the number and equations of tangential planes which deter­ mine enveloping polyhedron are computed, then the angles between planes and the locations of the polyhedron's vertices as intersection points of three or more planes. The results from these computations could be considered a mathematical model of the object. Then the program identi­ fies perpendicularity and parallelism between planes, if any exist. Having all these· results, the program confronts them with basic proposi­ tions, which could be assumed as the logical model of the object.

The second part of the program, which is realized only in principle but se far not yet written in computer language, provides, on the basis of confronting real and theoretical dis­ junctions, and by the use of basic propositional calculus, to obtain logical connections between basic propositions, with regar4 to the created object and to make decisions about the geometry of the object.

Exper1mental Model

In the first step of an experimental study, tactile information is taken by one finger with four sensors which are perpendicular to each other, as shown in Fig.6. The finger is able to keep in contact with the surface of an object, mDving along it.

The finger consists of a rigid support ter­ minated with a semisphere. On the surface of the semisphere, a number of small-pressure sen­ sitive on-off sessors will be distributed, according to geometrical criteria; however, this experimental finger is only distributed with four sensors, still according to geometrical criteria. A set of motors driven by electronic logic moves the finger in the space. The orien­ tation of the finger remains unchanged with respect to the three chosen axes: x,y,and z.

The surface equation is determined by the positiDn of the finger, when the sensor is in contact with the surface which delimits the object and by the identification of the sensor concerned.

If the finger is abLe to move, keeping in contact with the surface which delimits the object, it is possible to obtain the group of equations of the surfaces whose envelope ap­ proximates the shape of the object.

The block scheme of the electronic logic, which commands the control system of the finger, is shown in Fig. 7. The scheme shows the elec­ tronic concerning only one sensor for movements of the finger on the x and y surface. The elec­ tronic circuit is similar for all the sensors .

In Fig.7, a generoc sensor is symbolized by the switch S . . Si is closed, when the i-t h sensor is in cofitact with the object, and the finger moves along the object according to the predetermined direction of the switch s 1 . That is, a signal proceeds to block A, the essential element of which is an SCR. The voltage Vi, at the output of A, goes to the block B which con­ trols the motors M and M in order to move the finger in the dire~tion d~ associated with the sensor Si. Let us suppose that the sensor loses its contact owing to the S being opened. The block is such that Viis still present, to try to make new contact. We permit the finger to move the same direction di up to the maximum constant length: then we impose a movement towards the direction perpendicular to di, up to the maximum contant length if there is no contact; the following movement is imposed towards the direc­ tion, perpendicular to the last one, until the switch.makes contact with the object.

The movements of the 4-sensored finger are illustrated in Fig.8, and the following will happen:

1. contact is restored during di.
2. contact is restored at any moment during the cycle.
3. contact is not restored after the cycle

has been completed.

Fig. 9 shows the contour made by the 4sensored finger exploring the object. Thick lines represent the contour of the object when the switch is closed, and the fine lines the cycle as shown in Fig.B. The points indicated by x are the contact-points on the surface of the object. We can draw one contour line (thick line), if we provide an appropriate angled sensor on the finger. Wa are also considering using a ball with tactile sensors instead of the finger to recogni~e a surface in the same plane of the object.

Concluding Remarks

In this paper, we have discussed conceptu­ ally the principles of a visual-tactile symbiot­ ic system combining stereometric pattern recog­ nition with tactile perception involving one finger with 4 sensors. However, in computer simulation, we considered only the tactile per­ ception of grasping with two fingers and provid­ ed that the two tangential planes at the point of contact of the fingers with the object are almost parallel. For this purpose a terminal device with two fingers articulated in the pro­ portions of the human hand has been partly real­

ized.

In reali~ations of the symbiotic system, in order to incorporate the results of tactile per­ ception discussed here, we can start by arrang­ ing the explored area in a more convenient way, particularly adapted to the optical part of the system. We can divided this area with horizon­ tal pLanes distant one from the other for incre­ ment 6 z z 1. In our reference system equations of these planes will be:

If we remember that the general equation for tangential planes determining the envelope of a polyhedron is:

then the common solution for both equations will be:

i.e., in explicit form:

which will adjust the analytical exP£ession for intersecting lines between tangential planes and sets of parallel z-planes. If we form quadratic martices in these z-planes, and put L along in­ tersection lines and 0 on rest, we will have tation. Now we can uniformly consider optical

some kind of binary digitalized contour represen· and tactile information.

In the tactile model, the sequence of switches is an important for recognizing ob­ jects, in order to determine the directions of the finger's movements as shown in Fig. 8. If and only if in z-plane, the sequence of the switches should be ordered counter-clockwise as sl' 52' then continuously sr· sr· ········ srii2 where. n is numbers of switches in the same plane, and each switch provides the same angle on the surface of a finger.

From this introductory research, it is possible to point out that the visual-tactile symbiotic system is more practical and efficient for stereometric pattern recognitions than the use of conventional methods.

Acknowlegements

The authors would like to express their appreciations to professor E. Caicniello and Ing. S. Levialdi for their stimulative descuss­ ions.