Proceedings

Experience with ROBOT in 12 Commercial, Natural Language Data Base Query Applications

Download BibTeX Cite

Proceedings of the Sixth International Joint Conference on Artificial Intelligence. .

© 1979 International Joint Conferences on Artificial Intelligence

Abstract

£ne aoility to unaerstand ~atural Language has long been a goal of Artificial Intelligence rteaearch, and it is still far from being solved. rlowever, in toe early l~lv's the AI research techniques reached a point whereby certain a~plications became feasible for toe first time. Since that time, several ststems sucn as PLANES[!], LIFER[2], and ROBOT[3,4,S] have been built tnat nave oemonatrated tnat the current state of the art is sufficient for quite ~ooa natural lan~uage aata uase ~uery. ~ne i~plementation of the R030l' system has been ~eared for high performance and installaoility in actual real world environments. As sucn, it offers the AI researco com~unity aome inai~ht into toe aifficulties encountered when putting t~e c~rrenc AI tecnnolo~y in tne nands of people in the real world. This paper u1sc~sses cne unex~ecteo lin~uistic and semantic difficulties encountered in the 1~ co.n .•lercial applications to wnicn t<Jao ·r has been applied during the last year

Full text extracted automatically from the PDF; figures, tables and equations may differ. The PDF is the version of record.

1. i·H'~Juv-.:J.'J.,.h~

~ere-ace .nany differences oetween natural 1an~~a~e atste~a oevelopeu for researcn anu natural lan~~age atstems uevelo~ea for proa~ction. ~ot toe least of tnese oitferences is tne fact tnat toe prou~cc1on systems are event~ally useu in toe aosence of toe aevelopers of toe syste~. As sucn, valuaole insights into tne proole~ of natural lan~~age processin~ can oe gained. In spite of toe fact tnat all of us speak English, none of ~s Know exactly wnat the user is ~oin~ to type. deyond tnis, we don't Know now toe user's ~uestions will cnange after several illontns of use. Tois is valuable information tnat only ststeills tnat nave actually been installeu for long periods of time in cne user's environment can provide.

~ne ~0dJ~ stste~ has been a~plied t o 1~ ~ifterent a~~lications during tne pasc lo illOntns. In eacn case, toe SfSCeill oas oeen "turned over" to users ror e~a1uat1on. Toe ~nrase "t~rnea o~er" is Jargon of cne software inuustry .. teaning taat toe prvgram is now co~~letelt 1n cne nanas of toe ~sees. ... ·.a<> 1s ~acticulacJ.t oitficult to ao for nat~ral lang~age stste~s, since so ~ucn a~~llcation-specific tuning ia re~uired.

But the system must be officially turned over if we wish to observe the unencumbered user. Naive users, in particular, are greatly influenced when technicians or researchers are observing tneir use of a computer system. It is only after the system is completely in their hands that we can begin to gain accurate data on how well the system

satisfies their needs.

2. TH£ 12 APPLICATIO~S

-aoao·r has now been completely turned over to users at the following companies: digelow-Sanford Carpet, ~o~~ercial Onion Assurance, Du Pont Chemical, Planning Research Corporation, uunn ana draastreet and Software Ag of L~or tn America.

Tne following is a list of the different data bases to which ROdOT has been applied in these installations ay ROso·r development staff: Customer file, Homeowner's Insurance file, E~ployee relations file, Car file, Employee file, Data dictionary, Financial file and budget control; by local staff: Sales file, Membership file, Accounting file and Planning file.

rae variation in tne different discourse aomains of tnese applications illustrates the importance of the domain-independent parsing techni~ues employed oy K080f (see [S]). fhe sizes toe applications range from l,uuu records in toe test files to over 2.4 .nil lion records.

J. ACf0AL u~~rt rt~J0~df~

~ae --ro11ow1n~ ~uestions were

selected from lo~ f1les 1n response co a aurvet re~uesting interesting ~ueries c~ac nave oeen aaKea oy local users tnat rt~d~r was aole to answer. Altnougn tois saill~le 1s oiasea towaras re~uesta tnat rtvd~J.' can nanole, it aoes de.nonstrate coe level of ~upoist1cation tnat users exeJeCt tru,;t t·le ayste,n.

~dv~ ~udt~fAL~ vi Oirt~~f CO~~~ti~l~~ dt ·'JU·H'd f~rt .i.,:;J.' ..JJ.'rt 1 o r.~ rteui~.-. 1>1 ~ri~rte ~~f Art.i.~.i.~~ A~0u~£ i,; At LeAd~ .;il::iv,vuv.

i~rt uv~~~f.i.C MA~d.i.~~rti, ~KL~t rae JuL1 1~/d JALeJ ~vL~ArtS A~~ dALeS ..JUA~~!f!.c;~, d0rt~~u di SALeS CLAdS.

F~rt C~l-i·"'.c;rt~!AL u.i.ll!dlv·'4 Al~u .i.'d0.3~ rrtvuuCrd i~ ~4~~,:;.:; ~F .;i::ivu,uuv iBArt-£0uAte, ;jdv~ ··IE rtLL t!··I.C: .t>l::rtlvuS nrivdl!: .t>.K~- ·J.'A4 eArt.~.i.~'4~S Artt,; t-l0.KL J.'riAI-4 :;J.vv, uvu A.'4~ .t'rt.i.1'4J.' ·J.'de.i.rt t'4A··I.t:.

F~rt dt.JI'MArtlt:.:) rl!J.'d Ludd RA.i'.i.O ~rt.c;AJ.'.:ort J.'riA·i .l.uu, rte.t>u~d' rt.:oulv·-4, drtAt'4Cri AI~~ L~d~ rted.:ortlle di 1'1v·'H'd.

.t'rt.i.~r ALL .t'Kuuu~td rldude ~uve~d~rt l~tl ~A~I:O~ ..JUA~ti~.i.~d A~c d.c;.L'rJE.C:~ J..uv, uuu Al~U Jvu, UIJIJ rt.~u nd03e .i:>.KL::-·J.'A4 e.t\rt.~J.d...,,:, .c..o~h.:.:or.;~ .,;1, uuu, uuv.

'*. rrtvd~r.;,·t.::l .c;.~..:vu·'4J.'r;rtt.u

~c.1uu~a--~~-overa11 experience wJ.co avdv.L' ac alJ.. of cnese aitea naa ueen ~u1ce ~uu~, tnere ~ave nonecneJ..eas ~~en several vert ~1rri~u1t ~ruolems of a J..lo~ulacic or aemanc1c nacure coat ·let if~ a"'r ta~eo. J.'·h~ re,naHtoer uf toia ~aeJer ~1ac"'saea coeae eJrO~lems anu ~ucentldJ. so1uc10113 fur t.le.n. A proule.n 13 u~rine~ as ant oesirea res~onse co a re~uesc toac cannvc oe eJrvauce~ uy a~~ru~r1ace uict1onart oe£in1cions. ~oce coac tn1a 1s a ratner strict uer1oit1vn .;;ince ic eJre3Uutes t·lat tne ~araer ana tne set of primitives toat can ue useu to create oictionary oefinit1ona are fixeu. .t:ven toe sim~list cnan~e to tne At~ ~rammar is consiaereu a major proolem uecause tnis cannvt oe oone uy local staff at the site. A solution is aefineu to be a domain-independent extension to ROdOf and/or tne dictionary utility that allows tne desired response to oe proouced oy appro~riate dictionary definition. rne i~portance of tne domain-inaepenoence of toe solution snoulo ue clear from tne variety of applications tnat rt0d0f currently oeals witn. it is also clear tnat researcners wno ao not snare our concern for domain­ inuepenuenc solutions may find the following proolems very easy to solve in any ~iven application. Witn tnis uiscJ..aimer we can . no~ oegin discussin~ tne ~roolems known as:

Toe "~E 0rt I~" ~roblem, ~ne Snowmobile iroblem, rne Matrix Proolem, and

Tne du~nary Problem.

4.1 Tne "~~ 0K !~" .t>roolem

This problem ariSeS-as a result of coded data in the data base. The dictionary must contain enough information aoout tne coding scheme so tnat it can be made transparent to toe user. £ypically, users want to nave tne flexioility of referring to tne data in eitner tne !on~ form or tne coded form. For example, in a coded state field, tne user ~ay want to refer to ~ew rtampsnire eitner as ~ri or ~ew rlampsnire. Toe s~ecific proolem arises wnen a coded item intersects witn otner words. For exa~ple, tne aooreviation for ~aine is ~e, ana tne aoureviation of Jregon is urt ana toe aooreviation for Inaiana is I~. It is ~arc1cu1arly nard for tne parser to tell in su.ne uses of tne woru L~ wnecner its tne preposition "in" or Inaiana. rtudur normallt deals witn lexical amoi~uity oy consiaerin~ all interpretations and tnen neuristicallt selectin~ tne uest one. J.'nis approacn is fine in most circumstances, out does not ~ork very well wne one interpretation is mucn .uore co.nmon tnan toe otner. . For example, the re~uest "id!Nf Fu~ ME frlE ~AMEd 0F ALL deCrtetAK!ed." would oe detected as amoi~uous. Does it refer to all secretaries or just those in Maine? when you are aware of the information tne pro~ram nas to work witn, it's nard to fault it for clarification dialog cases like tnis. constantly bad3ered nelp weea out interpretations, it tir~so.ne. enterin~ into a witn tne user in aut wnen you are oy tne system to sucn obscure can ~et very

~~~ ~r~per ~o~~tion for tnis ~r~u1~ .. 1 .:>~~.ns t~ oe an idlproll'ea ne~rldtlc t~ac ld ~~rna~s a little illOre ru c.ue.:>.:i 111 weeuin-J v~c lncer):lrecations udlll-J cuueu uata ice..ns. unrort~nately t·lt! enll'lC~n .. lent ooe.,; nut ~roviae toe nor~al c1~e.,;, s~co a;; ca~italization, cnac ne1p ):leO):lle ffiaKe tn1s aistinccion. ~..L.:io toe Cf~ewrltten forffi of expres.,;ions tenus to exacecuate toe ):lrooleffi since ):leO):lle tenu to ~.,;e coueo references ratner fre~uentlf as a way of reaucin~ ttt>in~. ~ne current, aa~itteoly snort term, sol~ tion of rt0duJ.' is to restrict selecteo couea data iteffis so tnat they cannot oe ~sea as woras in a re~~est. ~nis allows tne person tnat o~ilas tne aicc1onary to oecid~ wnetner toe a.nui~uvus meanin-,J is aesiraole or not. t~is nas toe effect of forcin-,J users to refer to inuiana, ure-Jon ana ~aine 1n tne1r full forffi, wnere otoer states can ue referenceu eitner ~af.

... "' .. ·,ae .:»now.llouile .t'roole.n

--:;.;Ji.:> t;JC~·Jle.n-arose in toe context v.c cne ~ue.:i clvn "dOw ulany anuw.no.:>iles are coeret". ~n1s ~ue.:icion wa.,; aireccea cvwa.cu.:; a ·lv.neuwn.er in..>1..1rance .c ile t·lac lllC..LIJueu lntor ... atl~·l ~n anowi·lvulle ~ull'era~e. ~~e lncer~retaciun nvdJ~ ~ave tv ttll.:> re-11..1e.:it was "now .. 1any ):IO..Licies 11ave snuwulvui..Le cvll'era-Je'?". .~eeules.s to Saf, tlle a.tswers co tnese -al..lesclon.,; are ll'eCf ulfferenc. ~ne suoclett of tnis l"'rou1e .. 1 i.:iu't so .. 1ucn lin-:~uistic as it 1.:> 1n cne uaca cepra.;;entac1on. lf toe f1..1.e oe1n~ ~ueriea wo~lo nave containeo one recoro for ea~n snowffiooile, tnen rtvduJ.' woulo nave an.,;werea it properly. dut toe file containea one recora per t>Olict ana a tiela c~ntainln':J toe nuffioer of snow~ooiles coverea oy tnat policy. rtvdvJ.''s aictionary aefinea toe ffieanin':J or snowulooile to oe ant nu,noer .jreater tnan u in tnis field. Froffi tnis it snoulu oe c1ear wny .KJd0J.' first rouna cue ~o1ic1es wicn at least one .:illuw.,IOulle anu tnen countea t.le t;JO..l.lCl~-3.

~ne ~o1nc ia tnat tne coillpl..ltation ~.L t11e an.,;wer ue):lenus on now t11e uaca is .:>coceu. roe .;;yncacclc strucc1..1re of toe re~ue.sc 1.s ur llttle ne..Lp 1n I:l':JI..ICin~ wnac ~v~~ucac1on 1.:> re~u1reu to answer c~e re-1ue.:>c. ...:on.:ilaer c.ae rol..Lowin':J re~l..le.:it.:i:

".~uw ... ant anowul~ui ..I.e.:> are coer e?" "o~w ulaOj .:>ecrecarieS are t11ere?" ".tow u1aut .sa ..Lac .Le.:i are coere?"

These are all syntactically identical. One might even argue that tney are semantically homomorphic. But each request requires that the answer be computed in a totally different way. We have already seen that the snowmobile request requires summing the contents of toe snowmobile field. Assuming that secretary is an instance of a job title, tnen tne number of secretaries is computed by countin~ the number of records witn secretary occurring in tne joo field. Xne proper response to the request about salaries may well be to ~ive the nuillber of different salaries tnat occur in toe file. Needless to say, tne algoritnm required to compute tnis is very different from the otner "dow .nany" questions. .KOBOT is currently being fitted with enough domain-independent facilities to permit tne inclusion of doough domain-specific knowledge to know which algorithm to employ in each case ·

4. J ·rhe M.atr ix .Problem

Tnis problem--arises because there are many ways of referring to data aggregates. For example, one of tne applications had historical data in several fields such as profit and total sales. Thus, users could to refer to toe profit in l~ld or the 1972 total sales. In many re~uests the year specifier would refer to several fields. For example, "In 1972, wnat were the profits and total sales." In otner cases several years ffiignt apply to one field. For example, "~ive me toe total sales for l~ii ano l~iJ."

rnese ~ue~tions could not be processea oecause tney re~uire combinin~ toe syntactic inforffiation ~ainea from toe parse witn aoffiain-specific inforffiation aoout toe data oase. ~ne answers co tnese ~uestions are generated in oifferent ways depending on whetner toe years themselves are stored as data items. In tne particular instance in wnicn tnis prooleill was first encountered, .KOdOX re~uired very direct scopin~ of toe time referent. For example: "List toe 1~12 profits and the 1~12 total sales?", or "List tne 1~/2 total sales and tne 1~/J total sales."

Because tnis was perceived as a ratner severe lin~uistic liillitation, RJdJ£'s dictionary facility now allows tne specification of now tne tiffie referents and toe periodic fields are relateo. rnis information can tnen oe

1nce~racea ~ita tne syntactic 1riroe~ation ~eneracea oy tne parser to aecer~1ne toe set of fields oein~ reterencea. In fact~ one of tne act~al re~uest.s cnac appearea in tne ~ser survey re~~ireo tnid ~acility. ~ne ~nra.se "toe July 1~/~ sales dollars and sales ~~antitie.s" could not have oeen ~nuerstooa ~itnout it.

~.~ roe ~u~nary ~roolem

0 0 tne su~aary proole~ ar1ses because tnere .nay oe two uifferent ways to answer toe .same ~uestion witoin toe same oata oase. Xnis often occurs because a cerr.a1n a.no~nt of sum.narizatioo may occur as a aata oase is oein~ preparec roc ~~ery oy extcactio~ it from a lar~ec ~rouuct.ion fi~e. f~r exaffiple, if a file conca1n1n~ information on all sales tnat tou~< ~~ace in cne previous mont11 is to oe ~uec1eu, ic is often e~tracteu from a ~ac~ec fi~e conr.ain1n~ all sale.s. ~uc1n~ t.nls d~traction process, it may ue ues1eao~e co ca~culate certain du .. ~o .. lcHf d tdt.ldt.lCS anu ac.u..t tnem to tne r1~e. tnu~ a~l ~f r.oe tota~ sa~es may ue cvultlu teu foe eac·l .sa~ed.,lan ·

... 0 .1e ~r ~uleu1 occ~r s ~oen a pee son a~~<~ cur a t~lece ot s~funarizeo 1nt~e~ar.iun. toe exaffiple, "rlnat are tne coca~ sa~es for .saJ.es.nan d.;li to?" .L'oe uedleeu cesuJ.t 1s -iU.lte oowio~s. i'ne ~ro~ra~ anou~u ~erely t~rint. r.oe alreaoy co·t~uceu re~~lt racoer toan computin~ it a~ain fro· toe raw aata. ~oe reason t~at tois i~ difficult is oecause tne ~dera re~uest i.s inoepenoent of the fact toat tne a~ulular izeu infor.uation alreaoy e~l.st.s, anu tn~s toe ~arser ~enerates an lntee~retar.ioo for com~~ting tne total tru.n t·le ra~ ua.ta. !n tne exa.nple, tois woulu re~~1re retrievin~ all of toe uetail recorus for toe sa~esmen dmitn anu auuin~ ~p r.ne sale.:; affiounts. dince t~is ~o~lu oe ~ruo1oitive in large aata aa~e~, some ~ay of avoiuin~ tnis reca.Lculatlon mu~t oe rouna ·

  • .1£ toe uictlunary containea a ,:;eutant 1~,; ue,:;cr ltltlon or ~11ac su.n..1ar izeu 1orur·at1vo wad t~recalcular.eu, toen toe .;e,uautlc .structure r~c toe re..juest woulu ue ~u~~deeu a~a.Lost toese teffi~lates. ···len a .ilatctl occurreu, toe reteieva~ ot a d.LO~le r1e~u co~~... ee~lace toe cvul~utatlvua.L ~uc tlvn or t·le se.naot.ic .:iteucture r~e toe re~~est. ~nia a~~r~aco ~uulu ce~ulre toe uerJ.nltlon ot .:iuute relativelf COiiltllex ae.nantic structures w1to1n coe uictlonacy, since t.·le su .... uar 1ze1..1 inrormation coulu oe of

aroitrary dictionary be ennancea description complexity. ROBOor' s building facility will soon to allow tne semantic of su~narized information.

::) · 3l) L'i.'IA.rt :i

Xne R0d0X system is tne first natural ~angua~e system to oe put into proouction use in a variety of real worlu enviropments. rne problems encountereu in tnese applications are of interest to tne researco community oecause . in ·any cases tney coallenge toe funaamental principles on wnicn virt~ally all current natural langua~e ~uery SfStems are oasea.

rie nave presented four sucn pcoolems in tnis paper that were motivated oy actual field experience iliitn .rtOBO·r. As such ·, tney oegin to form a minimal specification for toe actual re~uiced. f~nctionality of a natural language oata oaae query system. Tnese proolems ace amon~ the first difficult issues tnat future systems must be able to solve if they are to oe of significant use in the real world.

  • [lj ~altz, o.~., "An Bnglisn ~anguage JUeation Answering System for a Large Kelational uata dase", CAC~, July 1~/d, vol il, no 1, p ~io
  • [Lj rtenorix, ~. et al, "ueveloping a oat~ral 1an~ua~e interface to complex aata", VLudJ ~roceeaings, 0ct 1~17
  • [Jj rtaccis, L.rt., "user Jriented data oase ~uery ~ito tne KJdvr natural langua~e ~uery system", IJMMS (1~/ ~ ) ~ . pp 0~ 1-llJ
  • l4J rlacris, L.R., "RvdJor: A dign ~ecformaoce ~at~cal ~anguage Interface for uata dase Juecy", TH 17-l, Matn uept, uartmoutn Colle~e

(::>J narris, L.t<., "usin~ toe uata dase !taelf as a Semantic Component to Aid in toe ~arsin~ of Natural Laog~age uata dase ~ueries", T.rt 77-2, Matn uept, uartmoutn Colle~e

  • [o] nino~rad r., understandin~ ~atu£al Lao~uage, Acaaeillic Press
  • [/j riooos, ~.A. et al, "Xne Lunar dcieoces ~atural Lan~ua~e Information ~ystem", dd'~ Keport 2J/d, June l~/2
Welcome Future Conferences AI Journal Division IJCAI Proceedings About IJCAI IJCAI Awards