Was the Title of This Talk Gernerated Automatically? Prospects on Intelligent Interfaces and Language

1. Introduction

Let me take it from far away. Language is the extraor­ dinary means provided by the human mind for commu­ nicating with other humans (and for structuring thoughts). Spoken language has been for a long time the means for communicating face to face. Written lan­ guage, a means for transporting language across space and time, was invented about 5000 years ago, most likely by the Sumers. At the beginning it was pictorial, after a few centuries cuneiform coding was proposed. Only much later, in the 13th century B.C. according to some archaeologists, was an alphabet first introduced (in Ugarit, in what now is Syria). It consisted of about 30 cuneiform signs and was, so to say, quickly recog­ nised as a breakthrough and was adopted by several peoples.

Meanwhile means for producing various instances of

a written "document" were invented soon after the first user does not know what information is available to her, or she may not have a clear idea of what she is search­ ing for. The need arises for systems that integrate a mediated information access paradigm and a naviga­ tional paradigm, where the user may use different mo­ dalities to explore the material. We believe that explo­ ration of an information space will become more and more a typical interactive attitude on the part of users, in a world inhabited by a multitude of available multi­ media information [Maybury, 1997]. All this is becom­ ing apparent with the current diffusion of the web and its various browsers.

Another key element of flexibility lies in the possi­ bility of a system of having a model of the user, in­ cluding her interests, idiosyncrasies and the dynamic aspects inferred during the interaction. This is instru­ mental for making sense of partial or not detailed re­ quests (or other acts) by the user, and for determining the system's actions.

A desirable feature is creating the appropriate pres­ entations of information: the relevant information is made available to the user at the proper level of detail, coherent with other pieces of information provided previously, and further exploration is favoured.

If information is to be presented in a flexible way, it is essential that an automatic processor does the job - in the case of text presentation, a natural language gen­ eration system. The latter is a computational tool that automatically "builds" a text (a sequence of sentences) starting from abstract (non-linguistic) specifications. Given the internal representation of the knowledge sources, the system decides what is the relevant infor­ mation to be communicated, it organises a coherent text structure and produces the most appropriate linguistic expressions to convey the message. Multimodal flexible presentations exploit synergistically the advantages that different media can provide in conveying the message to the user. In this case all processors must start from an internal representation and the system must organise media allocation and media co-ordination.

But our goal will be only partially attained if we do not begin to touch on the most difficult challenge of all: the challenge of keeping attention high, of building a seductive interface, of having the user be surprised and attracted by the creative attitude of the (artificial) com­ panion .... At the end we are very rapidly getting to the point where whatever message, whatever agent, what­ ever interface will have to compete over the most pre­ cious and scarce resources human beings have: time, attention and emotional involvement.

I will discuss in turn some of these themes, making reference to work we have developed at IRST.

2. Multimodality and exploration of infor­ mation

There can be different views on multimodal communi­ cation. Per se multimodality is multidimensional. Often it is only regarded as the combination of various uses of media, but certainly this is only one obvious aspect of the whole matter, that, besides, requires a clarification. "Multimedia" denotes the physical means via which information is input, output and/or stored. "Multimodality" refers to the human perceptual proc­ esses such as vision, audition, taction, and somehow it may also refer to the interpersonal or person-artefact context that develops in the interaction. Intelligent (multimodal) systems in principle tend to be character­ ised by a representation of the content of the presenta­ tion, so that presentation material is not fixed and can be customised dynamically.

Multimodality can also refer to the combination of various attitudes in relation to communication and in­ formation access (e.g. goal-oriented and exploration­ oriented), each one having in isolation its specific char­ acteristics.

Another view is concerned with the relation to human capabilities. Multimodality may amplify our capabili­ ties. In order to do so it must be cognitively compatible, but altogether it may not reflect our "natural" commu­ nication.

Yet another aspect is related to the specific role that language plays in the multimodal context [Stock, 1995]. It is a different role than when communication is based on a single modality, and so its operational characteris­ tics are different; its modelling requires specific com­ ponents.

Integration of NLP and hypermedia in a multimodal system offers a high level of interactivity and system habitability; each modality overcomes the constraints of the other, resulting in a novel class of integrated envi­ ronments for complex exploration and information access.

According to [Waterworth and Chignell, 1991], there are at least two dimensions for a model of information exploration: structural responsibility and target orienta­ tion. Structural responsibility involves the issue of which agent (i.e. the user or the system) is responsible for carrying out search and for giving structure to in­ formation. It gives rise to a dichotomy between naviga­ tional and mediated exploration. The dimension of target orientation presents a dichotomy between browsing and querying. Browsing is distinguished from querying by the absence of a definite target in the mind of the user. This distinction is determined only by the cognitive state of the user, not by her actions nor by the configuration of the system. In reality there is a contin­ uum of user behaviours varying between querying and browsing so that it is inappropriate to build systems that reflect this strict dichotomy, imposing one particular attitude on the user's exploration.

In work carried on for several years at IRST we de­ veloped an environment in which interaction could smoothly move along the two dimensions. Dialogue management then had to include a communicative ac­ tion co-ordinator, responsible for proper media usage (and so, for example, it can take into account the deictic context at any time of the interaction) and for suggest­ ing to the user a shift along the structural responsibility dimension.

Let me now summarise a system called AlFresco. For this system and for some other experiences I report here you will see a common application theme: cultural heritage and tourism. No wonder: Italy is considered to have half of the world's cultural tourism resources. Besides, the field can provide a wonderful opportunity for introducing technology that can help shifting from a mass-oriented attitude to an individual-oriented attitude - exactly what we aim at. Cultural tourism can become an experience where the individual is the active subject of the exploration, one who develops a personal taste and interest.

AlFresco [Stock, 1991] is an interactive, natural­ language centred system for a user interested .· in Four­ teenth Century Italian frescoes. It has the aim of pro­ viding information, and also of promoting other mas­ terpieces that may attract the user. Hypermedia is inte­ grated both in input and output. The user can interact with the system by typing sentences, navigating in an underlying hypertext, and using the touch screen in a coherent multimodal discourse setting. In output, im­ ages and generated text offer entry points for further hypertextual exploration. The result is that the user communicates linguistically and by manipulating vari­ ous entities, images, and text itself. The system builds a simple model of the user as the dialogue proceeds and uses it for output decisions, while allowing the user to browse around freely.

A higher-level, pragmatic component decides how to react in the given dialogic situation, considering the type of utterance by the user, the context, the model of the user's interest, the things already shown or said to the user and so on. The dialogue may cause zooming into details or changing the focus of attention onto other frescoes.

We have proposed a level of multimodal acts repre­ sentation [Stock eta/, 1997], roughly corresponding to what, for strictly linguistic dialogues, is the illocution­ ary level (Fig. 2). The key point for multimodal inter­ action is provided by the uniform use of felicity condi­ tions, the rules that govern the relations between inter­ actional exchanges and communicative intentions.

The dialogue cohesion management [Zancanaro et a/, 1993, 1997a] provides also a graphical feedback of the dialogue cohesion status to the user. This visual repre­ sentation: (a) reassures the user at a glance on the sys­ tem's interpretation (as such it takes the place of a para­ phraser), and (b) allows co-operative recovery from discourse misconceptions by means of a series of "in­ tuitive actions" when this interpretation is not the one the user meant.

In general a tighter integration of different modes of exploration, language-oriented and navigational, has been accomplished. I think this is a very fruitful con­ cept that can lead to various applications. The study of the involved cognitive aspects is of great importance and lab experiments with implemented prototypes and simulated systems will make all of us understand better this kind of amplification of human communicative capabilities.

3. Bringing physical space into the picture

One further element for advancing in the direction of personalisation and context-sensitivity is offered by ubiquitous information access, made possible by hard­ ware technologies such as portable devices and wireless networking. A museum is a privileged environment for introducing adaptive information with ubiquitous ac­ cess. In fact, the experience of visiting a museum typi­ cally consists in moving in a physical space and ac­ quiring information about the objects shown (and of course in becoming interested and moved by what is displayed!). In the new interaction scenario, the com­ puter (a hand-held device including spoken output) allows the integration between the physical space (through a positioning system) and the related informa­ tion space, yielding a new way of exploring cultural heritage. The individual visitor is at the centre of the physical-virtual space exploration and her movements and interactions provide input to the system to tailor appropriate presentations.

The approach presented here was developed inside a project at IRST called HyperAudio [Not et a/, to ap­ pear] . The results are at the basis of the development of an even richer interaction scenario that is being ex­ plored jointly with other partners in HIPS, a European project of the Esprit e program 1 ·

The problem of adapting content for (cultural) infor­ mation presentations in physical hypernavigation shares many features with the problem of producing adaptive and dynamic hypermedia for virtual museums [e.g. ILEX, Mellish et a/ 1997] or dynamic encyclopaedias [e.g. PEBA-11, Milosavljevic et a/, 1996]. Moving in a physical museum has been the goal of the RHINO proj­ ect, where a robot accompanies the visitor [Burgard et a/, 1998] .

Content adaptation in a physical environment poses some problems that are related to the fact that the visi­ tor is experiencing a "real" situation: the cognitive problems that may arise when a person is moving in a virtual information space, are different when the user is seeking concrete objects, moving in a real environment that provides stimuli, attention grasping and feedback. Information is presented in different situational con­ texts, determined mainly by: (i) user position and movements; (ii) the structure of the surrounding physi­ cal space (e.g., whether objects are close or not); (iii) whether other people are examining the same item or not; (iv) whether the user came alone or not.

1 The[HIPS] consortium includes: University of Siena (It­ aly, coordinating partner), CB&J (France), GMD (Ger­ many), IRST (Italy), SIETI'E-Alcatel (Italy), SINTEF (Norway), University of Dublin and University of Edin­ burgh.

HyperAudio (and HIPS) integrates the individual, dy­ namic modelling of the user with a general model of the environment, of the user's movements and of the dis­ course history to best tailor information presentations. Different forms of adaptation are introduced by the system, both in the information provided and in the further steps suggested. In general the approach points to a realistic and evolutionary adoption of generation techniques; at present it yields a rethorically coherent dynamic combination of small existing fragments of speech.

The architecture abstracts away from specific imple­ mentation solutions. It can be implemented on a single mobile platform (as in HyperAudio) or with some mod­ ules running on a standing platform and communicating with the mobile computer via a wireless connection (this solution is investigated within the HIPS project).

When deciding what information to include in the presentation and the most suitable discourse structure, the system takes into account various knowledge sources about the user and the interaction (Fig. 3).

' ·--······-····--·-·-··-··········--·-··-··-·-·-··-·-··

·· ...

Fig. 3 Architecture

The user model is accessed to exploit: a) the user's interests (which are inferred from her behaviour), to include in the new message information that can stimu­ late the hearer's attention, possibly proposing informa­ tion about other objects/sites strictly related to what the user is seeing, to increase curiosity and desire to ex­ plore; b) the user's background knowledge, in order to relate the new information presented to what she al­ ready knows (therefore reinforcing learning) and to decide whether additional clarification or exemplifica­ tion of new concepts is required to help her understand. As the interaction proceeds, the system refines its as­ sumptions about the user's interests and knowledge by observing the user's behaviour and by keeping track of the information she has been exposed to. Another knowledge source is the history of previous interaction. An important role in content selection is also played by discourse strategies that the system exploits to guaran­ tee that topics are presented in a coherent order and the various discourse chunks are linked by rhetorical rela­ tions that reinforce the understanding of discourse flow. The system consistently limits the length of audio mes­ sages, deciding to realise part of the content as click­ able links on the screen to avoid overwhelming the visitor with information.

The language style adopted for each single user is selected according to the user type.

According to information contained in the current context of interaction (e.g. position in relation to dis­ played objects, to the more extended environment, .. .. ) and in the discourse history (e.g. the topic of the previ­ ous sentence or presentation), the system selects an appropriate linguistic realisation for referring expres­ sions and spatial references. Other cohesion devices (like anaphora, conjunctions, lexical cohesion) are properly introduced to guarantee the fluency of the message and enhance understanding.

The system includes also a graphical interface that helps to orientate the visitor and is useful for comple­ menting linguistic instructions. Besides, "clickable" elements in the oral presentations appear on the screen.

Input provided by the user to the system can be both implicit, corresponding to movements in the physical space, and explicit, corresponding to interaction with the palmtop screen. Input is first analysed by the Intel­ ligent Input Analyser that decides on the most suitable type of processing required (e.g., plan a new presenta­ tion, stop the current presentation, plan a navigation support message, etc.). From all input the Behaviour Observer derives possible refinements to the User Model.

The Presentation Composer is responsible for plan­ ning an overall presentation that integrates (where ap­ propriate): object descriptions, images supporting de­ scriptions, buttons and menus for follow-up information requests, directions for navigation support and maps. Fundamental resource for flexible output generation is the Macronode Repository. Each macronode includes a network of message fragments (audio, text or images), a list of pointers to other relevant macronodes (specifying the particular rhetorical relations among them), the type of message (e.g., introductory label, caption, ... ), and a pointer to the relevant semantic concepts in the ontol­ ogy. The network of message fragments encodes the different ways in which it is possible to realise the content of the macronode and thus encodes its surface linguistic forms, while the relevant concept, the rhetori­ cal links and the message type encode the deep struc­ ture of the description.

Many of the issues presented for the museum setting apply to any physical hypernavigation setting in which individual, dynamic guides would be appropriate: for example historical cities, archaeological sites or natural settings such as gardens, parks or mountains. It is obvi­ ous that wide open spaces introduce additional options from the technological point of view (for example the adoption of a GPS as the localisation system) and sug­ gest more ambitious scenarios (for example: new func­ tions to support groups of visitors, access to on-line services such as meteo forecasts) . All that is subject to investigation inside the European I 3 Project HIPS. An­ other element being explored in HIPS is the adoption of global strategies for presenting information and pro­ moting items, making sure the visitor does not miss them. The simplest strategy is a gravity-driven one: there is a basic path where the visitor, through presen­ tations and suggestions is attracted, whatever deviations she performs; distances from a position to the next position tend to be minimised. Another one brings into the picture the dimension of play: for instance a dy­ namic treasure hunt, where typically physical distances tend to be maximised.

Yet another innovative feature is the introduction of collective memories. The visit trace is kept. The data can be used by the visitor: when she is back home she will be able to go deeper into the domain she has been exposed to, with a system (like AlFresco) that knows about her visit and will support her in her successive exploration.

Other possibilities are there for treasuring some spe­ cific itinerary (for instance one made by an art critic or by a public person) so that it can be followed with mi­ nor deviations by another visitor. Yet another opportu­ nity is to build models of the behaviour of classes of visitors and on that basis influence the curators' choices.

3.1. What about speech?

Speech processing is a key element for natural interac­ tion systems. I believe that synthesis in particular will prove even more important than recognition. Often the user's input can be very simple (as in HIPS) but still output, as it may depend on other implicit input or on a user profile, may require a lot of sophisticated proc­ essing for achieving a good presentation level. With personalised output often you want information to be presented as a coherent text, prepared for you and pre­ sented orally. Concept-to-speech (integration of gen­ eration and synthesis) is yielding good results, but even synthesis per se has improved tremendously.

Coming to spoken input, there has been a lot of prog­ ress in spontaneous speech recognition, albeit in highly constrained dialogic settings. For instance within the C­ ST AR II consortium IRST and its partners have built a prototype aimed at making possible that two persons physically remote and each speaking her own language [Cettolo et a/, 1999], entertain a conversation oriented to book a hotel room. Again the application is relevant for the tourism domain and translation is the most ap­ parent result, but probably the really important techno­ logical progress in input is in the ability to treat natural speech phenomena, such as false starts, hesitations and so on. Recently we have begun to work on spoken dia­ logue within a highly sophisticated, immersive, graphi­ cal system about cultural heritage.

4. A challenge - Collaboration

What described so far is not enough. Dialogue must be seen as a collaborative enterprise and we need models that help us understand multimodality at this deeper level. Our overall systems at IRST do not make use of this concept yet, but we have worked "in vitro" on this. We have considered the multimodal interface as a place where actions occur that may be considered both as domain actions and communicative (linguistic and non­ linguistic) actions. This is true for user and system actions: when the system holds the initiative it performs some domain actions, some communicative actions or actions of both kinds. The interface is at the same time a sensorial organ, the collection of media through which the message is realised, and the (virtual) place where domain actions are actually performed. In the old "teletype approach" this ambiguity was not present.

The intentional structure of discourse has been mod­ elled in [Lochbaum, 1999] . Her proposal emphasises the collaborative aspect of communication, by means of a peculiar kind of plans called SharedPlans. The theory of SharedPlans [Grosz and Kraus, 1996] is based on the notion of "plans as complex mental attitudes", and is intended to model interaction as a joint activity in which the participants try to build a plan together: the plan is shared in the sense that participants have a com­ patible set of beliefs and intentions. In this framework, communication is seen as the way in which agents agree on the various stages of the plan construction.

The difficulties in applying the SharedPlan theory to multimodal interaction arise from the double nature of the interface: some actions (especially the linguistic ones) are intended to augment the current SharedPlan while others are primarily intended to execute the re­ lated recipe, but at the same time, if these actions take place on the interface, in some way they contribute to the augmentation of the plan too. For example, if an agent is committed to do an action it must perform it and then inform the other agent of its execution: but if the effects of the action are apparent on the interface neither the explicit commitment nor the informing are actually necessary.

Any intelligent multimedia system requires a compo­ nent that exploits the context to make presentation deci­ sions (media selection, co-ordination, allocation, etc.) or to interpret multi-channel input [Maybury and Wahl­ ster, 1998]. In particular, given information that needs to be displayed to the user, a multimedia co-ordinator automatically builds a coherent and co-ordinated pres­ entation using a combination of available media. [see for intance Wahlster eta/, 1992] .

Following [Arens, 1993] any complex multimedia co­ ordinator needs to be built around a collection of mod­ els: a model of virtual devices, a model of the charac­ teristics of information to be displayed, a model of the discourse and the communicative context, a model of the interaction participants' beliefs, goals, attitudes, capabilities and interests. Input and output processes interact with the dialogue manager that maintains the discourse structure and ensures a coherent interaction between the participants.

An important point is whether action execution is ob­ servable (and in principle interpretable as desired) by the other agent on the interface. This depends on the ability of the multimedia co-ordinator to plan a mean­ ingful presentation with the available media. The mul­ timedia co-ordinator is instructed by the dialogue man­ ager as to the communicative intentions and returns the planned presentation to the dialogue manager. The dialogue manager in turn evaluates the expected effects on the other agent, and whether the case asks for further planning. For instance in case the presentation is not perspicuous enough, it may decide to plan a further communicative action (for example an inform action).

We have proposed a specific augmentation and exe­ cution process for SharedPlans that can accommodate our view [Zancanaro et a/, 1997b]. Two basic elements needed to find their place: a) a "local coherence" tech­ nique that could be combined with the higher level coherence of the SharedPlan approach that views com­ munication as a collaborative activity, and b) multime­ dia co-ordination.

In explorative information access it is more difficult for the system to recognise the user's intentions, as far as real world actions are concerned. The attentional aspect is more relevant; yet the intentional aspect can be fruitfully inserted as well . General strategies of ex­ ploration can be conceived, even if not every action on the part of the user can be interpreted at the planning level. Besides, some interaction fragments certainly can just be modelled as task-oriented. A flexible combina­ tion of a more "localist" representation and a collabo­ ration-based one can be appropriate.

5. Computational humour- Not so crazy

Interfaces must be seductive: they must attract and satisfy the user. This is particularly true of situations where the goal is not so much work productivity, but a reality that is a mix of entertainment, information and education: so called edutainment constitutes an in­ creasing large portion of what the computer can offer to our life. We know a major challenge for our society will be to find ways to improve education, not only institu­ tionally, but at all level of activity. And we know that learning can go well together with active entertainment.

The human mind likes communication and emotions. Personally I consider humour an essential part of com­ munication. The relaxation of inner censorship and the release of energy that derives from it produce an intense pleasure that tends to repeat itself in a favourable situa­ tion for giving raise to this phenomenon. Interactive and individual-oriented computational humour [Stock, 1996], beyond entertainment, per se important, will help all kinds of concept promotion and, in general, of learning. I am sure, among application areas, it will constitute the key to a number of children's activity and games: think of the children's ability in finding ambi­ guities and absurd meanings and the match they could find in a computer; simple humour on the part of the system can be a great resource as it helps to memorise errors and corrections; it will help develop social be­ haviour, etc. Even in a domain of active exploration, humour can help keep attention high, promote items and memorise what is seen.

While humour without restrictions is certainly "AI­ complete", there is work that shows the feasibility of introducing some elements in a system [see for instance Binsted 1996].

A restrictive view of interfaces, normal today, will eventually yield to a more advanced view, in which interaction with the user will be accomplished for in­ stance through an assistant, a critic or through a group of characters. The work of Barbara Hayes-Roth [see for instance 1996] and others has shown some possible ways. Humour, I believe, has to play an essential role. The characteristics of the interface will determine its potential on education and on society in general and we should not miss this opportunity. It is worth remem­ bering what Groucho Marx said of TV: "I find TV very educational. The minute somebody turns it on, I go to the library and read a good book."

Conclusions

Language processing has indeed a large practical po­ tential if inserted in a multimodal conception of the interface. There are different dimensions for the con­ cept of multimodality. One refers to the perception of different co-ordinated media used in delivering a mes­ sage; another one to the combination of various attitudes in relation to communication and information access (e.g. goal-oriented and exploration-oriented).

In this presentation I have taken a practical perspec­ tive and I have referred to some implemented proto­ types, mostly conceived for cultural tourism, a sector that I believe has a large potential. We have started with a system, developed some years ago, where inter­ action was based on the seamless combination of navi­ gation and dialogue. We have then begun to take into account the physical space, with the goal of producing a personal, mobile device for person-oriented guided visits in a physical museum or a town. Toward the end I have talked about bringing into the picture a deeper level of modelling multimodal dialogues, based on collaboration, and briefly discussed the relevance of seductive interfaces and humour.

What else do I want to say about prospects for real­ ising more useful and intelligent interfaces?

• It is important to introduce elements of a cogni­ tively compatible, qualitative physics in our systems, so that in our multimodal interaction the system can under­ stand what the user's representations and reasoning about space and time are and how she naturally conveys meanings in her communication . .
• Case-based reasoning techniques can be a very relevant resource for presentation systems, especially in the edutainement and cultural heritage domain. We must find appropriate ways of using this resource in the multimodal context.
• We need to make experiments, especially Wizard of Oz simulations and understand more about what is really cognitively appropriate. Most of the things we foresee do not exactly correspond to what we find in nature (face to face communication) or in written text. Our technology must not be intrusive, and on the other hand it may be different from other modalities we have known for a long time.
• Natural language generation has a large potential for improvement. One thing our community can yield is the next step after hypertext. A situation where text fragments are tailored and the reader/hearer may want to interact with the system so that the material is pre­ sented to her with a certain style, taking into account an increasingly large number of personal characteristics and behaviours. On our future radiosets, we shall have not only the loudness setting but also an interactive adaptation of the presentation to our context.

By the way, the answer to the question in the title is no. But the question may have a different answer when posed again in the not so far future.

Acknowledgments

I wish to acknowledge the contribution of all the people at IRST who have worked at the AlFresco, HyperAudio, and HIPS projects, with whom the ideas presented here have been developed. In particular, Massimo Zancanaro and Carlo Strapparava, beside all this, have also been essential for developing the collaboration-based multi­ modal dialogue work.