NLG - Natural Language Graphics

D. C. Brown, S. C. Kwasny, H. W. Buttelmann, B. Chandrasekaran, and N. K. Sondheimer The Ohio State University Columbus, Ohio 43210

The goal of the NLG project is to enhance person-computer interaction by using more than one mode of communication. The project uses a combi­ nation of natural language and graphics for both input and output. This allows development of practical, habitable systems suitable for users who are naive about programming, and encourages productive use of computers by more people. Mixed graphical and linguistic interaction could be an important tool in a variety of application areas.

A pilot system [Brown & Kwasny, 1977] has been developed using an ag60 plasma panel screen for output, and a touch panel and keyboard for "touches" and text. Organization of the system is heterarchical and based on inter-module message passing. The system is capable of drawing, eras­ ing, and naming lines, points, or circles, and of answering questions about screen objects and sys­ tem capabilities. A typical dialogue might be (where <T> represents a physical touching of the

screen):

PLEASE DRAW A VERTICAL 2 INCH LINE HERE <T>. PUT A POINT CALLED FRED HERE <T>. MAKE A CIRCLE WITH A TEN CM DIAMETER AT FRED CONNECT FRED AND (150,100) CALL THE CIRCLE BALL ERASE THE LINE FROM FRED ERASE FRED CALL THE 2 INCH LINE BAT. DRAW A CIRCLE NAME THE CIRCLE FACE WHAT DID YOU DRAW HERE <T>? a LINE called BAT WHAT CAN YOU DRAW? LINES, POINTS, and CIRCLES IS THERE A CIRCLE CALLED FACE? yes CAN YOU DRAW SQUARES? no

One of the modules in the system is the Lan­ guage Analyzer module (LA). This module utilizes lexical and graphical input modules to form an in­ put string which is processed using an Augmented Transition Network (ATN) grammar [Woods, 1973]. If necessary during processing, LA can retrieve stored knowledge using a Question-Answering (QA) facility embedded in the Knowledge Base module (KB). When language processing is completed, KB is passed the resulting case-based semantic struc­ ture. Using the MENTAL package [Shapiro, 1974], KB builds memory structures in a semantic network which is based on the work of Brachman [1976]. The network has attached procedures which provide default values, check net structures, and develop program-like structures which are passed to a Graphics output module. All textual responses output by the system are produced by a language generation module which uses an ATN and templates.

Current research centers on language analysis, knowledge representation and protocol analy­ sis, but our interests extend to system organiza­ tion, language generation, graphics, person-ma­ chine interaction, and the processing of 'drawn in­ put. For linguistic input, a robust analyzer is being developed, based on a uniform model of syn­ tax and semantics to handle both sentences which are grammatically acceptable and those which are not. A grammar is said to be complete in this sense only when it accounts for every input possi­ bility explicitly.

Future Knowledge Base research will concen­ trate on the homogeneous representation and manip­ ulation of many different types of information, such as structural, lexical, and functional. Currently, we are working on the process of ex­ tracting, from stored knowledge about an object, that information which will allow a picture of the object to be assembled and displayed. We require that the representation should initially be able to support both pictorial and linguistic generation, a~d, eventually, input in both modes as well.

Protocol analysis should lead us to discover adequate vocabulary and grammar and preferred modes of man-machine interaction,particularly ratios of graphical to language use and usage pat­ terns. Some information about semantics can also be gathered by recording user reactions to system

responses.

We believe that with a robust analyzer, mixed mode input and output, and a powerful knowledge base, Natural Language Graphics will provide a productive research and applications tool.