Human Control of Remote Computer-Manipulators

Introduction and Problem Statement

Remote manipulation up to the present time has been carried out mainly in nuclear "hot cells" and to a lesser extent undersea. Generally in these applications a direct mechanical or electro­ mechanical linkage continuously slaves the move­ ments of the mechanical hand to those of a handle guided by a human operator a relatively few feet away. The only extra-terrestial application has been the manipulator on the Surveyor spacecraft which was commanded in very simple discrete move­ ments to dig the surface of the moon and to posi­ tion experimental apparatus. General purpose man­ ipulators have recently been employed in production tasks, wherein the manipulator repeats a stored motion sequence "taught" it by a human operator who moves its degrees of freedom through the desired path to record the important points along it. None of these devices makes use of artificial intelli­ gence or even environmental sensing for automatic control. An extensive historical review of devel­ opments in remote manipulation has recently been

written by Johnsen and Corliss.l

Beginning with the experiments of Ernst 2 sev­

eral developments of autonomous robots have begun at MIT, Stanford University, and Stanford Research Institute3·4,S which have potential application to an unmanned f.lars mission. Completely autonomous devices will surely be practical for a wide range of recognition and control tasks eventually. It is the authors' contention, however, that for the im­ mediate future it will be economically expedient to send a manipulator or vehicle having a low level of artificial intelligence but supervised by a human

Thomas B. Sheridan William R, Ferrell Hassachusetts Institute of Technology

Cambridge, llassachusetts

operator on earth. It would transmit to him rela­ tively simple sense data and other information for his interpretation and for use as the basis of fur­ ther instructions. At lunar and nearer distances the telemetry required is entirely within the pres­ ent state-of-the-art.

Here generally it is believed that non-routine mechanical handling or assembly tasks too remote or too hazardous or scaled too large or too small for direct manipulation by man can best be controlled by a mixture of human and artificial intelligence. The problem is how man and computer should interact. This narer describes laboratory simulation experi­ ments involving human subjects together with compu­ ters in real-time control of two, three and seven degree-of-freedom mechanical manipulator devices. In these experiments the computer continuously rec­ ognizes simple patterns and effects control trajec­ tories to achieve relatively near subgoals, whereas the man intermdttently observes system states and sets subgoals. The paper also illustrates by exam­ ple the ways in which a human operator and a compu­ ter can efficiently complement one another in plan­ ning and executing task strategies.

Loop Delays from Signal Transmission, Coding and Process Dynamics

A continuous control system usually becomes unstable when loop delays exceed one half cycle at any signal frequency for which loop gain exceeds unity. In control from earth the speed of light poses loop delays (round trip) of l/3 second to synchronous Earth satellites, 2,6 seconds to the moon and at best several minutes to Mars. To this is added the delay for signal coding and decoding which may be nearly five seconds even for high pri­ ority signals in complex state-of-the-art vehicles like Apollo. Low resolution pictures from Mars \~ill take several minutes, high resolution pictures an hour or more. Process lags of inertia and viscosity can add further delay to remote mechanical opera­

tions.

Laboratory experiments in human control of simple manipulators (two-dimensional translation and grasp) have shown that with only visual feed­ back human subjects avoid the slow and erratic move­ ments that delay tends to induce by consistently adopting a move-and-wait strategy, whereby limited open loop moves (without feedback) are punctuated by waits of one loop delay time in order to gain feedback and confirmation or reorientation,6 Task completion time can be accurately predicted from completion times and measures of open-loop perform­ ance taken when there is no delay. Unfortunately, the move-and-wait strategy takes time and comple­ tion time increases linearly with delay. Other ex­ periments with loop delays in tasks having contin­

uous force feedback (one-dimensional contact pushing tasks without vision) indicate that high feed­ back gain can cause instability and that searching movements must be slow to limit impact forces. 7

These results suggested that when time delays are necessarily present the human operator should not control in a continuous closed loop but should serve instead as a supervisor or intermittent moni­ tor and setter of subgoals for an automatic or com­ puter-control loop.8 The dynamic movement of the manipulator would then be under control of a local computer eliminating stability problems. Moreover, the completion time would become far less sensitive to delay since the operator would command rela­ tively long segments of the task, reducing the number of waits for correct feedback.

Such a system is illustrated in Fig. 1. A laboratory simulation has been implemented consist­ ing of an American Machine and Foundry Hodel 8 master-slave manipulator driven by stepping motors through an augmented PDP-8 computer.

Han-Machine Console Interface

The human supervisor may issue his commands as strings of alphanumeric characters through a conventional teletype as in Fig. 2a (such commands we call "symbolic commands"). He may also indicate direction, magnitude, and time through moving a multi-degree-of-freedom joystick which may even have articulations corresponding to those of the manipulator (these we call "analogic commands"). Experiments by Verplank9 with a device shown in Fig. 2b suggest it is quicker and certainly more natural for the human operator to demonstrate to the computer certain movements or positions by the analogic mode than symbolically. This is true even i f the analogic control device has only on-off switches for each articulation so that the operator must himself adjust his arm and hody to achieve the position correspondence. Conceivably the analogic control could also take the form of a scaled physi­ cal model of the manipulator in a model of the task environment. The human operator's goals could then be communicated to the computer by his manual rear­ rangement of the model.

We have experimented >~ith sensinq and disrlay of hoth visual and tactile feedback. As one would expect, ordinary television feedback i s ~uite sat­ isfactory provided the human onerator can, at will, call either for a close-up or a comprehensive view. The close-up is needed for details of grasp or posi­ tion of tool relative to object, while the compre­ hensive plan or profile view shm1s the manipulator base or absolute reference frame, the mechanical hand and the maninulated objects all in relation to one another.

Even the crudest of tactile sensors such as a simple electrical contact is useful for automa­ tically stopping the arm motion if an object is touched. The manipulator arm has sufficient elas­ ticity to render collisions with fixed ob j ects harmless, and transient shocks are quickly damped out. We have also developed a prototype touch sen­ sor havina on its surface a deformable mirror which distorts a regular grid pattern. The human opera­ tor observing the reflected image through closed circuit television can infer the point by ryoint pattern of normal forces on the grinping surface of the maninulator hand. This device is shown in Fia. 3.

Command Language

In order to develop, on-line, the commands appropriate to a particular task from a small set of primitive commands, a simple comPiler called MANTRAN has been written by Barber.lO It accepts typed commands to the manipulator to move at a spec­ ified rate to a specified position or in a specified direction until certain specified conditions occur, allm1ing the operator to combine these into "pro­ grams" that can be called by name. Such programs may have within them various searching or grasping or emergency conditional subroutines. A simple but useful mode is to be able to assign a name to a giv­ en manipulator configuration (simultaneous set of relative positions or angles at each articulation) and at same arbitrary time later to simply t ype the configuration name and have the manipulator auto­ matically return there. This requires, of course, that the computer maintain a state vector for the current manipulator configuration with which to cam­ pare the stored state vector corresponding to the named configuration. Alternativelv, the s ystem can be commanded to respond to the analogic controller until certain positions are obtained or until the hand contacts an object, at which time the program branches out of the analogic mode. An example of MANTRAN is sh01>1n in Fig. 4.

Functional Organization of the Computer-Maninulator Svstem

It is convenient to represent the breakdown of functions in the computer manipulator system as in Fig. s. Whereas Fig. 1 illustrates the nhysical location of major components, Fig. 5 indicates the various kinds of logical processing necessary for sur>ervisory control. Host but not all of these functions are implemented in the laboratory s ystem described above.

Beginning at the left, the ill-defined runc­ tion of evaluating and setting subgoals is perform­ ed by the human operator. He interacts with the symbolic (keyset) and analogic (local physical mod­ el) cont·rollers to convey his intentions. A com­ puter element called the command interpreter con­ verts these human responses into a uninue concaten­ ation of commands, or if certain criteria are not met, returns to the operator via the display with a query or a reminder.

The onerator, and in turn the interpreter, can give four kinds of commands to an executive contcoller: 1) he can instruct the exteroceptors (receotors which sense the external environment) to he positioned, oriented and focussed in a certain wav and to tell him what they sense, 2) he can request that certain "what would happen if " ex­ oeriments be conducted within an abstract represen­ tation of the manipulator and task internal to the computer; 3) he can request the execution of certain moves--simple ones like move manipulator angle A plus-two degrees, or complex ones like search region R for part P and assemble it with 01 4) he can re­ quest that certain transformations available within the computer system be applied to data and that the results then be displayed to him.

The block labelled "task. model" is itself worthy of far more discussion than can be given it in the present paper emphasizing the nature of the human operator's supervisory control, and its in­ ternal organization is discussed much more fully in a separate paper by Whitney submitted for this conference.ll In summary, however, one can say that if the positions in six-dearees-of-frecdom of all rigid elements of both manipulator and envir­ onment are represented by a single vector, then any interactive movements of the manipulator with the environment can be represented by a series of vectors or by a trajectory in vector space. In theory, then, optimal control strategies are those for finding minimal cost trajectories from initial state to goal state in the state space.l2

While a finely reticulated state space for a system having many motor states, many sensor states, and many environment (manipulated object) states is clearly far too large to be useful for overall control, the state space model provides a formal norm or baseline to which other techniques can be compared. State space models of a tractable size can be used to perform simple manipulations, or to perform parts of more complex ones. In any case an important practical problem concerns the definition of what are rigid elements, what -are costs, and what discrete resolution is appropriate to represent the real world in the state space model.

A simple example of interaction of the human operator with the task model is described in a sub­ sequent section of the paper.

It is important to note that the executive controller oversees many "automatic" feedback. loops and interactions which are not under direct control by the human supervisor. As indicated by dotted lines and letters in Fig. 5, the executive control­ ler mediates direct or straight through reauests for (a) sensory analysis of the environment or for (b) task. model experiments. Imperative move com­ mands automatically call for, in sequence, (c) search and identification of objects in the envir­ onment, (d) task. model experiments, and finally (e) manipulator movements.

The executive controller mediates (f) the feedback. from the exteroceptor actuators (actually the interoceptors of the exteroceptor subsystem, analogous to the position sensors of the muscles of the eye!) and the (g) signals from the task. model to control where to look. next. Similarly (h) the feedback control of the experiments of the task. model or internal representation of the manipula­ tor as well as (i) the feedback control of the actu­ al manipulator are handled by ~e executive control­ ler.

The task. model is updated by (j) the sensed state of the environment and (k.) the sensed state of the manipulator. Knowledge of (1) the sensed state of the environment and (m) the results of the task. model experiments are automatically used in optimal or sub-optimal control of the manipulator. A significant part of this latter function is the decision as to what individual angular changes of actuator movements within the manipulator will achieve the final desired position. The reverse transformation from component angles to gross configuration is easy trigonometry but the reauired transformation from joint or hand position to com­ POnent angles can re~uire complicated matrix inver­ sions, or approximation methods, or worse yet (in the case of redundant degree-of-freedom maninula­ tors) , be undefined. This specific problem has been the subject of several previous papers.l3,14

Finally the operator can ask for a variety of information (n) to be disnlayed to him. The oraan­ ization of this aspect of the system is probably very im_nortant but, since as yet no experiments have been done emphasizing it, it is left indeterminate.

Hote that the executive controller, the task model, the command console interpreter and also the display boxes must all be implemented laraely by digital logic. Which computer functions belong to the "local" computer and which belong to the "re­ mote" computer are unsolved engineering problems and will surely depend upon, among other thinas, the specific task. context, the amount of telemetry processing required for signals of the necessary precision and the like.

Human Intervention in Computer Control Procedures

Because of the economic limits on multi-dimen­ sional state snace or other formal models of whole manipulation tasks referred to above, a nrimary qoal of our research is identification of the ways human operators and computers can interact in plan­ nina and executing manipulation tasks. This ~rill be presented by example.

Fig. 6 represents a manipulation task in a reticulated 10 by 10 unit space. The manipulator ja~rs J and J' and the blocks A and B are each one unit on a side and all moves of blocks and jaws must be an integer number of units in extent. The goal is to move the jaws so as to get block A to square A'. Neither blocks nor jaws can occuoy a cross­ hatched square (wall). The jaws move horizontallv as one, and vertically J' moves relative to J over four spaces from closed to wide open (they are illus­ trated one space open). The jaws can grasp a block or push it; in order to grasp a block the jaws must first straddle it with one unit space on each side.

Table 1 gives an arbitrarv set of primitive functions and tests which can be concatenated in various ways to perform tasks such as the one in our example. The functions and tests are organized into three subroutines GRASP, MOVE and PUSH. Within each subroutine a series of tests is made: for a ~ answer the arrow to the right calls for an action and a return to the previous test or a jump to a dif­ ferent subroutine; for a no answer the downward ar­ ro~r calls for the next indicated test or action. In moving a block. from one position to another a general rule is first to try to move bv grasping the block., and if there is no room for that to try to move by pushing the block..

The task of getting A to A' is still too dif­ ficult for the subroutines of Table 1 to handle, and reauires the human operator to intervene at least often enou9h to break the task into three elements: move B to B', return jaws to left side of wall, move A to A'; by intervening a bit more often a con­ siderable saving can be made in use of subroutines.

A completely different procedure is for the human operator to provide the defining information for a state space, as referred to earlier in this paper and as described formally in a companion paper submitted for this conference by Whit­ ney,ll,l2 Given three objects (two blocks and a set of jaws) which can move to any position on a 10 x 10 grid, a vector space of size l02·l02,lo2 is required. Since J' can assume one of four states relative to J we need a total space of 4·106 states, We assume that an initial and terminal state (that is, initial and terminal positions of jaws and blocks in the 10 x 10 grid) having been specified, an algorithm exists for getting from one to the other in optimal or at least satisfactory fashion, details of which are given by Whitney.

If the operator intervened and called for a series of three initial to terminal state trajec­ tories (B to B1, return jaws to left, A to A'), then the state space required for each subtask would be considerably smaller than that required for the whole task, For this example B to B' would reauire at most consideration of all jaw states and ali B states but no A states, thereby reducing the state space a factor of 102, Return of the jaws would require no A orB states, netting a reduction by 104, A to A' would net a reduction by 102, Clearly, the sum of these state spaces, even if all of the 10 x 10 physical grid is allowed for each task element, is much smaller (4·104 + 4·102 + 4. 104 = 8·104) than the 4·106 state space reauired for the whole task, If, in addition to judging that certain objects can be eliminated from certain subtasks the human operator also judges that cer­ tain regions of the 10 x 10 grid can be eliminated from consideration (for instance in returning jaws from the B' neighborhood to the left side of the wall) then further reduction can certainly be made.

To make our ooint about the computational saving which accompanies intelligent intervention by the human operator permit us to digress from our example of Fig, 6, Suppose (Fig. 7) that for a 10 x 10 grid two objects X and Y must interact. At the outset suppose we (the human operator) know that A and B must interact in a general way within the bounded area 1, then must pass together from area one to area two while located t~ithin the 1-2 overlap area, interact within area 2, similarly pass to 3, interact in 3, pass to 4 and interact in 4, It would not be necessary to consider a state plane of si:ze (10·10) for object A cc:rnbined with (10·10) possibilities for B, or 104, This is because A and B are never simultaneously in differ­ ent zones except when together in the overlap areas, Thus a state space of (5oS) · (5·5) accomodatinq all A-B combinations in zone 1 overlapping a state space (6·5) · (6 .5) in zone 2 similarly overlapping a (6·6) · (6.6) zone 3 state space which in turn overlaps a (5·5) · (5·5) zone 4 state space pro­ vides for an optimal trajectory to be selected from among all possibilities. The total state space is 3446, less than 35% of the 104 state space, In general, a task involving n objects, each of which can assume any· one of s states, requires a state space of sn, Breaking the task down into m sub­ tasks in the ith of which only ri of the objects can move and each is limited to 0 i states renuires a state space of:

This will, except in unusual circumstances, renre ­ sent a savings in both computational time and storage, since ri ':!: n and Oi s S.

It is our ~urpose here to compare the savings in calculation and storage1 i,e,, the reduction in the total number of states, resulting from speci f i­ cation and delimitation of either heuristic proced­ ures or formal state spaces :--we-presume that a human operator is the one who intervenes, sets ap­ nropriate goals and calls appropriate heuristic routines, or that he bounds the state space--at least we do not understand how a higher level com­ puter program can do it.

P~turning to our first example, Table 2 shows various elements of the task of Fig, 6 , Task ele­ ments 1 through 11 are possible parts of the whole task (move A to A') and are not mutually exclusive, Task element 1 (move B to B') incornorates the rule "try movinq by grasping before trying nushinq," This procedure would automatically call subroutines in the order 1 2 1 2 3 and utilize 19 moves. For comparison, if the operator saw that pushing should be tried first, only heuristic 3 would be used and recruire 10 moves, The same objects are required fo~ the state space procedure in either case and we allow (conservatively) a 4·102 ·102 space, In either case ~~e state space procedure can be expected to calculate that a 10 move push is the best,

Task elements 3 and 4 are simpler elements, so simple that a human operator might be expected to call for the actual point to point move path (designated m) instead of calling a subroutine, vli th ~i ther of these the grid space required could be qreatlv restricted,

. Task element 5 could be included as part of 6 1 7 or 8 though here it is given senarately, Ele­ ment 6 actually illustrates a cascade of four sub­ tasks (move A to A1 1 then to A2, then to AJ, then to A'), The subgoals are achieved at the slash marks in the subroutine sequence, This series of heuristics would result if a human operator ini­ tially thought a shortest path was A to Al 1 then to A2, then directly to A', But after gettina to (or committing the program to) A2 he would realize (or have to have pointed out to him) that with A at A2 he has not left room for regrasping A and will have to set a new subgoal AJ to which A can be pushed and then grasped or pushed to A', If the human operator had thought ahead he would have set A4 and As as subgoals, and as indicated a shorter strinq or heuristics would be necessary, Again, by state space procedure, the same (but a very large) state space would get from A to A' automatically,

As a further example of how the operator can delimit the state space, suppose for task element 8 he recognized that from A1 ~ A4 to A2 ~ Jl.5 the jaws would have to be closed, Then the state space for this portion of the task need consider onlv block A o~er a 7·2 grid and the jaws J over a 7·1 grid, Let the jaws and A operate open or closed­ j awed over the grid B wide and ten high at the left for the A to A1 or A4 subtask and a over a grid five wide and ten high at right for the last part, This then renuires a . state space (4·3·10·3·10) + (7·2·7·1) + {4o5ol0·5ol0) = 13698 in size, This compares with the 40000 state space otherwise need­ ed, The situation is similar to that of Pig, 7, The task elements 10, 11, 12 and 13 provide a finer breakdown of the A to A' subtask.

Below the double line in Table 2 are shown several ways of accomplishing the whole task; in each case are shown the total string or heuristic subroutines and implied moves using that nrocedure; the total state space is also indicated with the implied moves for that method.

Indices of efficiency for heuristic or state space techniques such as these can be formed by making a ratio of the number of moves, number of routines called, or size of state space for a given procedure, to the corresponding number for a norma­ tive heuristic or brute force procedures such as the first whole task procedure given. Note that the last and most efficient procedure re~uires the human operator to observe that block B may be push­ ed out of the way by block A without need to modify the A to A' procedure. Much more dramatic efficien­ cies can be demonstrated for human intervention in the state space techniques when the ohysical space is more finely reticulated due to the

rule stated above, Heuristic techni~tes are by nature context dependent and not much generaliza­ tion can be made,

Conclusion

• A laboratory simulation and some empirical evidence have been described which demonstrate the advantages of having human operators operate as supervisory controllers of remote computer-manipu­ lators, where the man sets subgoals and sets pro­ cedural constraints and the machine does the rest automatically, Many practical advantages of such man-machine interactive systems are foreseen to speed up remote or monotonous tasks, avoid the costs and risks of supporting man in hazardous environ­ ments, and provide a true flexibility of explora­ tion and manipulation not near to achievement by completely autonomous machines.