Instructions: Language and Behavior

1 Panel Summary

This panel has three aims: (1) to inform the IJCAI-93 audience of the wide range of research in which instruc­ tions play a major role; (2) to explore some of the issues raised by trying to create effective instructions and/or to use them as intended; and (3) to link these issues with those of concern to mainstream AI. In short, we hope to make the audience aware of the benefits to be gained through work at the interface of language and behavior.

The panelists and their colleagues cover diverse parts of this range. (For pointers to other work, see Section 7.) Grosz and her colleagues are concerned with instruc­ tions in the context of multi-agent collaboration; Hirai and his colleagues are investigating instructions in sup­ port of telerobotic control; Rist and his colleagues at DFKI are concerned with the design and presentation of instructions using multiple media - creating and coordi­ nating effective presentations automatically; Scott and her colleagues are addressing a major problem of the European Community - satisfying the need for quality instruction manuals in all the languages of the Commu­ nity; Webber and colleagues at Penn are striving to use instructions to direct the behavior of animated human figures.

It is clear from the panelists' statements that there are several issues of general and immediate concern:

• Representation how to represent the content of in­ structions, such that the representation is adequate for negotiating or guiding behavior, for generating multimedia presentations, or for producing quality texts in multiple languages;
• Modelling how to model the agents receiving instruc­ tions, in order to choose instructions that they can understand and/or carry out as intended;
• Performance Monitoring how to establish that the recipient of instructions has understood them and is following them appropriately and without error.

Shigeoki Hirai ETL Tsukuba, JAPAN hirai @etl.go.jp

Donia Scott University of Brighton Brighton BN2 4AT, UK donia.scott@itri. bton.ac. uk

Instruction Quality how to ensure that instructions are coherent, intelligible and informative, and reflect what the recipient should indeed undertake to do.

This is the first panel we are aware of to address instruc­ tions from the viewpoint of AI, and we hope that it will benefit all those who take the time to listen or to read.

2 Instructions and Collaborative Plans

Barbara Grosz

Harvard University, Cambridge MA, USA

Cooperative problem solving by teams consisting of both humans and computers requires collaboration and communication. Our recent work has been directed at developing models of collaborative activity [GK93; GS90; LGS90] and at determining the types of infor­ mation about actions and action relations provided by certain natural language constructions [Bal92b; Bal92a; Bal93]. Our model of collaborative plans, SharedPlans, specifies the beliefs and intentions needed for a group of agents to perform together a complex activity that decomposes into a combination of individual and multi­ agent actions. Although originally intended to provide the basis for the intentional component of discourse structure [GS86], the model has been extended to cover multi-agent collaborative behavior more generally.

Instructions and SharedPlans have a two-way rela­ tionship. Knowledge of the way in which to perform an action is an important component of SharedPlans as well as of individual plans. We use the term recipe to refer to this type of knowledge; in particular, a recipe for a is a specification of a group of actions, which we will refer to as /3i, the doing of which under appropriate constraints, Pi, will constitute performance of a [Pol90; LGS90]. In this framework, instructions are a means by which one agent communicates recipe information to another. But also, this communication is an instance of collaborative behavior for which SharedPlans provide a model.

The definition of SharedPlans [GK93] requires appeal to two kinds of intentions, intentions to perform an ac­ tion (Int. To) and intentions that a proposition holds (Int. Th). In a SharedPlan for a, the agent who will perform {J; has an Int.To perform {J;; all other agents in the collaboration have an Int.Th this agent succeed. An "intention that" (Int.Th) leads to helpful behavior, but more specifically to an agent's intending to (Int.To) do an action 1 if it believes its doing 1 will help ensure the achievement of its intention that (Int.Th) the other agent succeed in doing {J;. The provision of instructions is one such 1·

Instructions may be provided under a range of inter­ active situations. The particulars of the situations affect the kinds of actions in which the instruction giver may engage to satisfy its Int.Th. For example, the instruc­ tions provided in a typical construction kit are a one­ shot deal. The writer must anticipate all possible com­ municative needs, and address them in the instruction sheet. There is no opportunity for the writer's Int.Th to engender spontaneous new Int.To's. In contrast, instruc­ tions provided interactively while an agent is performing a task are an instance of full collaborative activity. The agents involved engage in a SharedPlan for communi­ cating recipe information; new Int.Th's and Int.To's are generated continually.

We are currently examining the ways in which Shared­ Plans and individual plans interact in dialogue [Loc93], developing axioms of intending to and intending that, and investigating the interaction between SharedPlans and multi-media (natural languae and graphics) com­ munication [MR90].

References

• [Bal92a] C. T. Balkanski. Action relations in ratio­ nale clauses and means clauses. In Proceedings of COLING-9£, Nantes, France, 1992.
• [Bal92b] C. T . Balkanski. Actions, beliefs and intentions in rationale clauses and means clauses. In Proceed­ ings of AAAI-9£, San Jose, CA, 1992.
• [Bal93] C. T. Balkanski. Actions, Beliefs and Intentions in Multi-action Utterances. PhD thesis, Harvard University, 1993. Forthcoming.
• [GK93] B. J . Grosz and S. Kraus. Collaborative plans for group activities. Proceedings of /JCAI-99, Chambery, Savoie, France, 1993.
• [GS86] B.J . Grosz and C.L. Sidner. Attention, inten­ tions, and the structure of discourse. Computational Linguistics, 12(3), 1986.
• [GS90] B.J. Grosz and C.L. Sidner. Plans for discourse. In P.R. Cohen, J.L. Morgan, and M.E. Pollack, editors, Intentions in Communication. MIT Press, 1990.
• [LGS90] K. E. Lochbaum, B. J. Grosz, and C. L. Sid­ ner. Models of plans to support communication: An initial report. In Proceedings of AAAI-90, Boston, MA, 1990.
• [Loc93] K. E. Lochbaum. A collaborative planning ap­ proach to understanding subdialogues in conversa­ tion. Technical report, Harvard University, 1993. Forthcoming.
• [MR90] J . Marks and E. Reiter. Unwanted Conversa­ tional Implicatures in Text and Graphics In Pro­ ceedings of the Eighth National Conference on Ar­ tificial Intelligence, Boston, MA, 1990
• [Pol90] M. E. Pollack. Plans as complex mental atti­ tudes. In P.R. Cohen, J .L. Morgan, and M.E. Pol­ lack, editors, Intentions in Communication. MIT Press, 1990.

3 Language Instruction for Teleoperational Control

Shigeoki Hirai ETL, Tsukuba, JAPAN

My colleague, Tomomasa Sato (Univ of Tokyo, JAPAN) and I have been conducting research on lan­ guage instruction for teleoperational control of remote robot tasks. Key features of our approach are as follows: Language-based master-slave manipulator sys­ tem. Our research started with the idea of introduc­ ing robot lan~uage to construct a master-slave manipu­ lator system [SH87] to perform tasks such as chemical experiments in a hot cell. With this architecture, we hoped to combine the advantages of direct motion con­ trol with those of language specification. For example, the system could superpose language--specified motion constraints on direct control from the operation of the master-manipulator, thereby realizing human-robot co­ operation at the motion level.

For example, "(SORI '(A B) '(0 0))" commands the robot to keep rotation angle of its hand around X and Y axis 0, which means that the robot keeps its hand horizontal. This function is useful when the robot carries a container filled with water. The operator has only to concentrate on where to put the container, while the robot takes care of keeping its posture unchanged so as not to spill the water.

Teleoperational task-oriented knowledge base. Even in teloperational task execution, we need a vari­ ety of information about task objects, i.e. their handling properties and procedures for using them. It is quite nat­ ural to accumulate object handling knowledge that can be used when needed. It suits the aim of the telerobot, which is used for unexpected tasks in unstructured en­ vironments. We have constructed teleoperational task­ oriented knowledge base [HS88].

For example, the system takes care of procedures re­ lated to a "pick-and-place" operation, including one for managing object-location data, calling teaching proce­ dure if they are not filled yet, etc. This allows the oper­ ator to command the robot to operate an object, without the operator needing to check that all the necessary data is available beforehand or to update object-location data afterwards. The knowledge base also guides typical pro­ cedure for specific operations such as striking a match, igniting an lamp, etc. An appropriate teaching method specialized to use shape features of the object is typical example of guidance from the knowledge base.

Language instruction as behavior models. A world model serves as a resource for task execution based on language instruction. But direct intervention through master-manipulator may bring about differences be­ tween contents of the manipulator's world model and the actual environment. We introduced motion understand­ ing system to solve the problem [HS89] . In this system language instructions such as (APPROACH obj ect-x) or (GRIP object-x) give conceptual frameworksofbehav­ ior model for telerobotic object handling. For example, the GRIP instruction can be used when the robot holds nothing in its hand and the object to be grasped is in the middle of its fingers. The robot can then close its fingers. In the definition of GRIP are rules that interpret the slave manipulator's motion guided by the operator, and the interpreted results are used to maintain world model automatically.

Instructing other teleoperational functions than manipulator. As a word has several meaning, so has an instruction to a telerobot. The issues that we addressed above are primarily concerned with telerobotic motion. Supervising a remote robot means not only giving it tasks but also observing its task progress and results. We have developed an intelligent camera work control for monitoring remote robot tasks along with the task execution [Wa92]. One interesting point here is that the same instruction has different effects on the robot and on the camera.

For example, instructing the robot (APPROACH obj ect-x) makes the robot-hand approach the speci­ fied object. At the same time, it controls the camera to observe the entire path of robot motion, i.e. from cur­ rent hand position to object's location. Thus the same instruction affects the devices in different ways.

Communication with telerobot through lan­ guage. From the viewpoint of information flow between human and robot, research topics in telerobotics so far have concerned mainly with sending information to re­ mote robots, through instructions from human to robot . We recognize that information flow from robot to human is also important. We expect robots to be able to report not only task specific information but also any other in­ formation about the remote environment, on the request of the human operator.

For example, if the robot falls into trouble in the midst of a task, the operator may want ask "What's the mat­ ter? What's the cause of the trouble?," and hope the robot to answer as" An unknown (not described in the program or world model) object obstructs my way. I cannot find how to remove it."

Communication with telerobot through language will be quite interesting research topic.

References

[SH87] Sato, T. and Hirai, S. Language-Aided Robotic Teleoperation System (LARTS) for Advanced Tele­ operation, IEEE J. Robotics and Automation RA3(5), pp.476-480, 1987.

• [HS88] Hirai, S. and Sato, T . OBJECT MODEL FOR TELEROBOT, Proc. IROS '88, Tokyo, 1988, pp. 701-706.
• [HS89] Hirai, S. and Sato, T. Motion Understanding for World Model Management of Telerobot, Proc. ISRR, Tokyo, 1989, pp.124-131.
• [HS89] Hirai, S. et al. Intelligent and Cooperative Con­ trol ofTelerobot Tasks, IEEE Control Systems Mag­ azine 12(1), 1992, pp.51-56.
• [Wa92] Wakita et al. Automatic Camera-work Con­ trol for Intelligent Monitoring of Telerobotic Tasks, Proc. IROS92, pp. 1130-1135, 1992.

4 Exploiting Multiple Media for Instructions

Thomas Rist Elisabeth Andre DFKI, Saarbriicken GERMANY

There are several good reasons why most instruction manuals for technical devices appear as illustrated texts. Pictures show where parts are located and how things have to be manipulated. Pictures can also disambiguate referents via visually perceptible properties. Texts on the other hand enables us to abstract from irrevelant details and allows concepts to be conveyed, such as it­ eration and conditional action, that are hard to depict using static graphics alone.

Computer-based multimedia presentation techniques provide more effective means in instructing people in task performance as they are able to overcome prob­ lems arising from the non-adaptive, static and non­ interactive nature of conventional technical documenta­ tion. However, instruction with multimedia presentation techniques is not advantageous per se. You might con­ sider your own experience as regards badly crafted man­ uals in which illustrations may, rather than facilitating comprehension, even confuse readers.

Designing comprehensible instructions for human users that minimize execution time and errors is a chal­ lenging task. Questions that arise in the context of mul­ timedia instructions include:

• What should be conveyed by text, what by graphics or other media?
• Which kinds of cohesive links between media are necessary?
• When should information be redundantly encoded by 'paraphrasing' it in different media?

A lot of work has already been done in diverse disciplines in order to give guidance to writers of technical manuals and instructions. To a certain extent, rules of thumb, such as 'use medium a for information /3', have been proven useful. However, in order to set up a sophisti­ cated theory concerning how to design instructions, we have to spend much more attention to an instructee's perceptual and cognitive processes. For example, when selecting a presentation medium, one has not only to take into account the nature of the data and the medium, but also the inferential processes a certain medium sup­ ports. As computer scientists we can offer new presenta­ tion forms as well as means for simulating both physical actions and cognitive processes. This includes both theo­ retical work on computational models and practical work on concrete systems.

The 'Intelligent User Interfaces' group at DFKI started in 1989 with the project WIP (Knowledge-based Presentation oflnformation) [AR92; Wa91 ; Wa93]. WIP aims to develop a presentation system that transforms formally represented domain plans in comprehensible in­ structions for human users by exploiting different presen­ tation media such as text and graphics. The generation process is controlled by a set of generation parameters such as presentation objective, target group, resource limitations, and target language. The benefit of WIP lies in its ability to present the same information in a variety of ways depending on the value combination of the generation parameters.

The project 'PPP (Personalized Plan-Based Presen­ ter) [An93J continues work done in the WIP project by adding three fundamental extensions. In contrast to WIP, the 'PPP system not only synthesizes multimedia documents, but also plans how this material is to be presented to various users. Furthermore, 'PPP allows for user interaction, i.e., 'PPP responds to follow-up ques­ tions concerning the domain as well as to meta com­ ments on the act of presentation. Whereas WIP has no means of finding out whether a user really understood an instruction, 'PPP monitors the effectiveness of a pre­ sentation. The 'PPP system is able to keep track of the user's behavior and continuously adapts its instruction strategies to the current situation as it is based on a physical connection to the technical devices which are to be manipulated by the user.

References

• [An93] E. Andre, W. Graf, J . Heinsohn, B. Nebel, H.-J. Profitlich, T. Rist and W. Wahlster. 'PPP- Person­ alized Plan-Based Presenter. DFKI D-93-5, 1993.
• [AR92] E. Andre and T . Rist. The Design of Illus­ trated Documents as a Planning Task. In M. May­ bury (ed.) Intelligent Multimedia Interfaces, AAAI­ Press, 1993. (DFKI RR-92-45, 1993.)
• [Wa91] W. Wahlster, E. Andre, W. Graf and T . Rist . Designing Illustrated Texts. Proc. of the 5th EACL, pp. 8-14, Berlin, Germany, 1991.
• [Wa93] W. Wahlster, E. Andre, W. Finkler, H.-J . Prof­ itlich and T. Rist . Plan-Based Integration of Natu­ ral Language and Graphics Generation. AI Journal 26(3). (DFKI RR-93-02, 1993.)

5 Multilingual Generation of Instructions

Donia R. Scott

University of Brighton, Brighton, UK

Instructions are an important application area within Europe, where there is an increasing demand for instruc­ tions for products and processes to be available in all the languages of the Community. However, current practice for the production of commercial multilingual documen­ tation hinders the efficient writing of effective instruc­ tions insofar as product/process development, authoring and translating tend to be three discrete and sequen­ tial processes, making use of different information and involving different skills:

• domain experts Software product developers write product-oriented functional descriptions of the soft­ ware system; developers of bureaucratic processes write texts written in a highly formulaic, legal­ istic language. Their skills are primarily techni­ cal/bureaucratic, not linguistic.
• technical authors Professional writers of instructional documents use the information provided by the do­ main experts to write user-oriented instructions. Their skills are mainly linguistic and graphical rather than technical. They tend to be monolin­ gual.
• translators Translators work with the instructions pro­ duced by the technical writers and aim to produce an equally successful set of instructions appropriate to a different linguistic readership. Although trans­ lators tend to have similar skills to technical au­ thors, they must be bilingual. While they should be good writers, they are not always so; they are mea­ sured primarily by their bi- or multilingual skills.

Because of the discrete nature of the processes, the fi­ nal instructions tend to suffer from problems of drifting specifications 1 and of 'Chinese whispers' 2 · Furthermore, the different specialisations of the two types of docu­ ment producers very often lead to difficulties in produc­ ing good quality translations of the instructions: original texts are often (at best) badly written or (at worst) in­ complete, misleading or even incomprehensible.

One approach to this problem is the use of a controlled language for the original instructions combined with the use of human or machine translation (the text-to-text approach) .

We adopt an alternative approach- that of multilin­ gual generation of instructions from a common represen­ tation of the underlying task. In two new collaborative projects involving academic colleagues in Austria, Italy, Spain and the UK, and groups of professional technical writers and translators of commercial and bureaucratic instructional texts, we are developing monolingual tools for (not necessarily bilingual) linguistic experts. The tools are to provide equivalent first drafts of instructional documents in selected languages of the European Com­ munity (English, French, Italian and German) and in two application areas (user manuals for software prod­ ucts, and instructions for bureaucratic procedures, e.g.

1 the lack of a common perspective among the three partic­ ipant groups within the production process, combined with the lack of clear or complete initial information, resulting in documentation which does not meet the requirements of the user.

2 the distortion and loss arising from misinterpretation, re­ duction or elision of the information that the translator re­ ceives from the technical author.

how to apply for a farm subsidy or for unemployment benefits).

A strong motivation for eschewing the text-to-text ap­ proach comes from the observation that texts are neces­ sarily selective, embodying choices of expression appro­ priate only to the requirements of that one language; for the automatic generation of equivalent instructions, a linguistically predetermined representation cannot be the primary input, since the choices made for one lan­ guage cannot, in general, be undone for the purposes of another.

Our multilingual drafting tools focus on two of the problems associated with the text-to-text approach for multilingual instructions [DHS92; DSH93]:

• discourse structure Different languages often place different constraints on the presentation of informa­ tion in instructional text . This is reflected in varia­ tions in clause ordering, discourse segment ordering and the ways that the relationships between clauses and segments are conveyed.
• clause level considerations Different languages often impose markedly different requirements on the ex­ pression of any given fact . There are two sources of difficulty here. First, syntactic equivalence of constructions does not guarantee pragmatic equiv­ alence. Second, different languages often do not re­ quire equivalent pragmatic markings in the expres­ sion of the same information.

The approach of multilingual generation provides the flexibility required to generate congruent, pragmatically equivalent, texts from a common underlying representa­ tions of the actions or procedures the reader is expected carry out. Clearly, this is important, since different lan­ guages often present instructions from different perspec­ tives. A typical example of this can be seen in the follow­ ing excerpts (taken from instructions for a step aerobics machine) which have the same underlying goal of getting the reader to balance the apparatus. Note that in the English and German versions the reader is informed how to balance the apparatus, while the French reader is told why the milled wheel needs to be turned. These differ­ ent perspectives lead to different choices at the level of discourse structure, which in turn impact on choices at the clausal level.

The apparatus can be balanced on uneven floors by turning the milled wheel.

Durch Drehen der Randelmutter kann das Gerit unebenen Bodenverhiltnissen angeglichen verden.

By turning of the milled wheel can the appa­ ratus (to) uneven floor conditions adapted be­ come.

Tourner 1'6crou mollet6 pour adapter l'appareil aux in6galit6s du plancher. Turn the milled wheel to adapt the apparatus to the unevenness of the floor.

References

• [DHS92] J. Delin, A. Hartley and D.R. Scott. Com­ parative Pragmatics: Discourse Structure in Mul­ tilingual Document Generation. Internal Research Report, Information Technology Research Institute University of Brighton, 1992. · '
• [DSH93] J. Delio, D.R. Scott and A. Hartley. Knowl­ edge, Intention, Rhetoric: Levels of Variation in Multilinguallnstructions. Submitted to the ACL-93 Workshop on Intentionality and Structure in Dis­ course Relations, Columbus OH, June 1993.

6 Instructions and Human Figure Animation

Bonnie Webber Norman Badler U of Pennsylvania, Philadelphia PA, USA

The research being carried out in the AnimNL Project ("Animation and Natural Language") aims to enable people to use instructions to guide animated human fig­ ures through a task. The figures come from an anima­ tion system called Jack™, developed by the Computer Graphics Research Lab at the University of Pennsylva­ nia. Jack provides articulated,· animated human figures capable of realistic motion through model-based behav­ iors [Ba93]. Jack agents can be anthropometrically sized and given different physical capabilities. In addition, different spatial environments can be constructed and modified at will, so as to vary the situations in which tasks are carried out.

Our ultimate goal is animated human figures that behave as semi-autonomous, instruction-understanding agents whose lowest-level behavioral goals can be fed into the simulator that sits above Jack, to be integrated and realized as realistic and informative animations. The potential uses of such a capability are many: from human task analysis, with tasks communicated through Natu­ ral Language, to small group training, where the other "members" of the group are animated human figures.

The problems are attempting to solve come at several different levels.

Language. The system must be able to interpret in­ structions in terms of (1) intentions that an agent should adopt, (2) desired constraints on the agent's behavior, and (3) expectations about what will happen. Doing this is not always straightforward. For example, Di Eu­ genio [Di92; DW92] has shown how purpose clauses af­ fect what action an agent chooses to undertake: when told to "fold a square in half to create two triangles", an agent will perform a more specific action - folding the square along a diagonal - because only by doing so, will s/he create two triangles.

Planning. In response to instructions, the system must make symbolic decisions about how an agent should try to achieve the intentions it adopts. Again, this is not always straightforward. For example, when an agent is warned about something, it must adapt its behavior so as to comply. One way is by just avoiding the action being warned against . ("Don't leave your room.") But more complex types of warnings ("Be careful not to cut outside the pattern lines" , "Do not adjust the switch plunger to engage with maximum force", etc.) require more complex and indirect behavioral adjustments.

Behaviors in AnimNL must be achieved through an integration of incremental, hierarchical planning and re­ active planning. The reactive planner is one that can directly "sense" the environment and control the loco­ motion of the agent directly. This obviates symbolic or global numerical path planning to approach a target, while avoiding collisions with nearby objects [BB93].

Two other planning problems that we have encoun­ tered are (1) that naturalistic behavior seems to demand that agents take account of future intentions when decid­ ing how to satisfy current ones [Ge92; GW93]; (2) that objects are not always to hand or in known places when needed, so an agent must then look for them [Mo93].

Obviously, these are general issues of planning and rational behavior, independent oflanguage, but the need to act appropriately in response to instructions makes the need for solutions more pressing.

Body. Some problems lie at the level of the agent's body- how an agent with a given body size and strength, and with the resources it currently has available, should move itself in order to achieve its intentions. For ex­ ample, an agent must decide where to stand in order to open a door to let someone in, or how to position itself and bend in order to reach a ball that has rolled under­ neath a table [Ba93]. Even for apparently simple "grasp" actions, considerable reasoning is needed: gaining con­ trol over an accessible object requires that one determine an appropriate approach path, grasp, and grasp site to achieve task intentions.

Integration. Finally, some problems have to do with how levels are connected such that current and future intentions can propagate downwards and be used to gen­ erate natural body movements, while performance in­ ability or failure can propagate upwards to affect subse­ quent planning and re-planning decisions. Coordinating the performance of multiple simultaneous and sequential actions through simulation is a non-trivial control prob­ lem. As for performance failure, whether they can be handled at the motor control, reactive planning, or in­ struction understanding level may depend on other fac­ tors. In particular, we must understand the influence of the agent's inherent skills, the coordination of multiple simultaneous task actions, the role of previous task per­ formance, the process of tool selection and use, and the use of assistance from other agents.

Contributors to the AnimNL Project include Tripp Becket, Barbara Di Eugenio, Chris Geib, Moon Jung, Libby Levison, Michael Moore, Michael White and Xin­ min Zhao.