Intelligent Multimodal Stream Processing

1. Intelligent News on Demand

Figure I illustrates the BNE and BNN systems that inte­ grate components from MITRE, Oracle, Carnegie Mellon University and Lincoln Laboratory to process imagery, audio, and text to enable news on demand. Key elements of BNE include scene change detection and classification from imagery, silence and speaker change detection from audio, and named entity extraction from speech or closed caption transcripts. A correlation component extracts cues from across these streams to detect the start and end of broadcasts, commercials, and stories. Subsequently, stories are classified and summarized, including selection of key frames, named entities, and representative sen­ tences. Finally, the user can search and explore video stories, provide relevancy feedback, specify media pref­ erences, and obtain personalized displays using the BNN web browser as shown in Figure 4 below. As detailed in [Maybury 2003], BNE and BNN transform video access from sequential to direct search, providing novel navigation and discovery mechanisms such as topical and entity-specific news clusters. Empirical evaluations have shown that users can find stories and answer questions nearly three times as fast as searching digital video with no loss in precision and recall.

Analogously, as shown in Figure 3, BNE classifies 15 frames per second into images associated with program start (e.g., logos), commercial breaks (e.g., blackframes), single and double anchor shots, and reporter shots.

3. Search, Retrieval, and Visualization

After BNE segments stories with multimodal cues, BNN supports the retrieval of stories by source, date range, keyword, named entity, or topic query. As exemplified in the upper left of Figure 4, after selecting news programs (e.g., CNN NewsNight, ABC World News Tonight) and indicating date ranges (e.g., Februrary 6-9, 2003), a user types in keywords in the text box. If a user is unfamiliar with retrieval terms, they can select from an alphabetic listing of all the named entities extracted for the time period and from the programs of interest, such as shown in the person, organization, and location lists in the upper left of Figure 4 (note "Hans Blix" is highlighted). This retrieves 14 matching stories and displays them as a "Story Skim", the source and date, the top 3 named enti­ ties in each story, and a representative key frame from each selected segment. Finally, by selecting a story, the user is presented with "Story Details", including all named entities, a text summary, and access to video and transcript sources. In empirical studies this mix of ex­ tracted media and hypertext organization enables analysts to perform searches three times faster than with video source.

Figure 5 illustrates a user visualizing named entity fre­ quencies and animating story occurrences associated with geospatial regions over time enabling information discovery.

Acknowledgments

This research involved collaboration among Stanley Boykin, Andy Merlino, Warren Grieff, and Chad McHenry, Chris Clifton, John Aberdeen, John Burger, David Day, Lynette Hirschman, and Rod Holland.