Visual Analog Representations for Natural Languages Understanding

Abstract

In order i'or a natural language system to trul y "know what it is talkin g ahoul ,"it must havt· a ·· oniH '<" tion to th•• r ea l-world correlates of lan g uage. l'or lan g ua ge d esc rihin )', physical ohjects and their relations in a scene, a visual analog representation of the scene ca n provide a u sef ul target stru ct ure to he shared by a lan g uage understanding sy st e m and a co mputer vi . sion system. This pap t> r discusses th e ge neration of visual analog repr e s e ntations from input l·: nglish senL<'IH't'S. IL also dt·scrihes th e op e ration of a I.T S I' program whi c h generate -s su c h a r e pre se ntati'"' [rom s imple l·: nglish sentences d e scrihing a scene. A sequ e nce of sentences ca n r es ult in a lairly e Laborat e model. The program can th e n answer questions ahottt r e lationships betwe e n the ohje c ts, even though the relationships in qu e stion may not have been explicit in th e ori)~ illal s cP nP des c ription. R<·sults suggest that the dir ec t testin g of visual analo g r e presenta1 ion s ma y he an important way to by pass long c hains of reasoning and t.o thu s avoid the co mbill; ll i'"'''' prohl<'ms inlwn•nt in su c h reasoning methods.