Introduction

Narrative theory¹ suggests that people are essentially storytellers and have an implicit ability to evaluate a story for consistency, detail, and structure. From an analyst perspective, a story offers a common form of communication for investigating the feasibility of a connected collection of characters and their motives. For these reasons, a story narrative presents a potent way of capturing the analysts' insights that can promote sharing of observations and understanding of complex phenomenon. Beyond communication, a story allows people to build spaces in which to think, act, and talk in a framework.² In this sense, a story serves as an internal communication tool – a method of organizing the daily observations into meaningful knowledge. It is the ability to pull information together into a coherent story that guides the organization of observations into meaningful structures and patterns. To an intelligence analyst, a story acts as an artifact of tacit and gained knowledge applied to raw observed data.

The prototype GeoTime Stories system introduces three sets of features intended to allow analysis at a higher level of abstraction than traditional visualization tools. The first is a space–time pattern finding system that relieves the analyst from the need to search for some common behaviors and relationships among events and entities. Found patterns are passed to an annotation system that visually highlights the pattern events with an expressive graphic. The annotation system also tags the internal data model with information about the annotation. Finally, a specialized text editing panel is provided to allow analysts to author narratives and explanations (reports) containing links to bookmarked views of events in time and space. The text system also provides a means to categorize content to support collaboration or multiple story threads. This system was developed following a cognitive task analysis of analysts. Informal reviews of the prototype system with analysts suggest that this system could increase the descriptive capability and clarity of their insights and reports. A formal evaluation was then performed by the National Institute of Standards and Technology (NIST) with experienced intelligence analyst subjects to test authoring and asynchronous communication of intelligence reports using GeoTime Stories. Analysts rated most GeoTime Stories capabilities between high and highest for functionality, utility and usability. An example Stories-based report generated by one of the subjects is included in the section Formal Evaluation and Development. Another example of an analytic story, based on non-Stories GeoTime functionality was published prior to completion of the capabilities described here.³

Background and related work

A story or narrative is essentially a means of communicating a sequence of events related to the experience of characters. Plowman⁴ clarifies that although both terms narrative and story are typically seen as synonymous, a narrative specifically refers to the macrostructure of a document whereas a story refers to both structure and content. In Plowman, the narrative macrostructure was considered more problematic and more worthy of investigation than supporting content exploration. In intelligence analysis, however, the content of a story is also of particular importance and they both need to be equally addressed.

Classically, Aristotle, in Poetics, describes the objects of a tragedy (or story) as plot, character, and thought. Thought is described as the processes of reasoning that lead characters to behave as they do. Plots are 'arrangements of incidents' and tragedies are most importantly the representation of actions, not of characters. However, plots do more than organize events into a coherent structure. The authors of tragedies, or poets, grasp and represent the internal logic in a story that underlies the necessity of the outcome of the events. Laurel⁵ states that a story offers a context to understand and communicates activities and plots played by characters or actors. Context helps an audience apply their tacit knowledge in order to better comprehend the point being communicated by the author. When applied to analysis, a story can therefore be a very useful medium for clearly communicating a situation and related information.

Various approaches to connect information visualization with the communicative power of storytelling have been investigated, each with advantages and drawbacks in the context of its usage by intelligence analysts. A common technique is to use hypertext links to multimedia⁶ or additional textual^{7, 8} content. The LifeLines project allowed for the incorporation of multimedia and interactions for creation and playback of connected temporal data,⁹ using interactively linked 2D displays. The MyLifeBits system incorporates a lifetime's worth of media to tell a story.¹⁰

In Vaucelle,⁶ video content can be retrieved and associated with text in a system called a Textable Movie. This paradigm allows the author to assemble pre-recorded video clips together to form a movie based on the text content. The focus of the Textable Movie system is to add narrative structure to existing content, rather than address both content and structure. In systems such as the Textable Movie, linked content is immutable and limited in its context. The system is limited in that links are intended to be viewed but not manipulated and investigated further.

Zellweger⁸ presented another interesting visualization technique for the hyperlink story through the Fluid Reader. It has been observed that stories should be more or less linear with minimal context switches to connected information;¹¹ an apparent contradiction for hyperlinked documents. To address this issue, Fluid Text includes links that expand in place to allow the text to still be read linearly. This is advantageous for providing footnotes and additional context on demand. The concept of a dynamic content story that can be read linearly is very interesting; however, it is not amenable to an important requirement for some analytical techniques: the ability to compare and contrast different versions of stories that share some key aspects. Analysis of Competing Hypotheses is a common analytical method in which multiple possible explanations for an observation are explored. In order to support multiple story threads and collaborative analysis, the system should be able to remove, mix, and categorize content across the entire document. For example, if different characters tell parts of a story, hiding one character's contribution could significantly change its meaning. This is more like a filter on the story content, rather than a link to other content.

An important exploration of the social aspects of collaborative analysis combined with annotation on visualization is Senus.us.¹² Senus.us uses a blog-style discussion workflow as the basis for communicating analysis. GeoTime Stories takes a different approach; it is designed for authoring and presentation of a complete story, which is better suited to the types of reports generated by analysts who are experts assigned to specific tasks. In addition, the annotations made by Sensus.us are implemented as vector graphics that are overlaid above a 2D visualization, whereas in GeoTime Stories, annotations are located in time and space as 'data-aware' objects; they are semantically connected to the underlying data and become new items of information. This makes it possible to trace an information path from a sentence in a story to the actual event data that it refers to. Many-eyes.com¹³ offers a similar discussion blog-based system. However, instead of providing graphical annotation tools, it allows users to highlight data points in a linked view. This method does allows specific data to be semantically linked to a comment, but does not provide for graphical expressiveness.

Finally, it has been suggested that visualization itself can be augmented using animation, voice narration, and visual annotations to tell a story about analysis.¹⁴

GeoTime – Unified temporal and geospatial analysis

The Stories system is built on the GeoTime time–space visualization framework, which is designed to improve the perception and understanding of entity movements, events, relationships, and interactions over time within a geospatial context.^{15, 16} GeoTime supports representations for the base elements of a story: events, people, objects, places, and relationships. Its ability to display these elements in time and space makes it a suitable platform on which to build representations and interactions with stories. GeoTime represents events within an X, Y, T coordinate space in which the X and Y plane represents geographic space, and the Z-axis represents temporal space.

Analyzing observations over time and geography is a common task but typically requires multiple, separate tools. The objective of GeoTime is to enable the analysis of information connectedness over time and geography within a single, highly interactive 3D view as shown in Figure 1. Events are animated in time through the 3D space as the slider bar is moved. A previous evaluation of GeoTime found that its unified geo-temporal representation increased analysts' understanding of entity relationships and behaviors.¹⁶ GeoTime allows the perception of who and what in the where and when.

Figure 1.

GeoTime for analysis of events in time and space.

Full figure and legend (408K)

Design and context

The objective is to enable analysts to perform analysis at a higher level of abstraction than currently possible. Rather than working with raw or atomic data, the analyst should be able to work and think in terms of behaviors, events, and plots such that it leads to comprehension, discovery, hypothesis generation, and communication. In preparation for the GeoTime Stories investigations, a brief cognitive task analysis study was completed including structured interviews, examination of work place artifacts, and a review of the literature.^{17, 18} From the structured interviews conducted with 14 analysts, it was clear that the idea of 'story' is broadly applicable to analysis. Deciding which dots to connect and how, what the context is, what the alternatives and the implications are, how one story leads to another, and how to share observations and collaborate are daily and critical tasks. The following are illustrative interview excerpts:

• 'Coherence and narrative are principles that guide the organization of observations into meaningful structures and patterns ... Plots are formed of dominant concepts or leading ideas that the analyst uses to postulate patterns of relationships among the data ... a story must form a logical and coherent whole and be internally consistent as well as consistent with the evidence.'
• 'It's good to put all the factoids together. See how they relate to hypotheses. Trajectories of factoids over time can tell a story.'
• 'A hypothesis is an assertion. An elaborate hypothesis, a story.'
• 'Once you have the data observations, and can see the links, in time and geography, you need to go to generation of hypotheses. [You need] to think of new possibilities and new lines of inquiry.'

A theme emerged that suggests incremental hypothesis generation and the need to tie a story back to evidence was particularly important. A simple model of story emerged, as shown in Figure 2, that helped to conceptualize the relationship between story concepts and the analytical workflow. The model defines a spectrum that ranges from low-level facts and details to higher level organizational structures, such as character and intent. Frequently used analytical concepts were located within this space. There are two workflows represented in this model, roughly described as bottom-up and top-down. Visualization and pattern detection within raw data serve as bottom-up methods for building information into higher level concepts. In contrast, a top-down workflow occurs when analysts apply their tacit knowledge to guide the search for information and patterns in the raw data from experience. The role of visualization in this model is to spark rapid recognition of those internalized stories by expressing patterns of activity in an easily digestible form. Bridging between experience and raw information is the basic activity of analysis. It enables prediction by mapping the known facts onto the familiar plotlines that an analyst has previously seen. We seek to facilitate this process in GeoTime.

Figure 2.

The story concept spectrum as conceived for stories.

Full figure and legend (153K)

Using the story spectrum model, two different approaches were defined to close the gap between the analytic stories and data visualization. The first was to help the analyst recognize patterns by creating higher level visual pattern representations. This should stimulate faster recognition of tacit stories in the mind of an analyst. An early concept sketch, shown in Figure 3, explored the use of expressive annotation to summarize complex, raw data for rapid recognition by the analyst. The second approach was to work from the other end of the story spectrum to support the analysts' application of tacit knowledge and organizational structures. The system should enable analysts to capture thoughts, and then allow them to bridge from thoughts to patterns and all the way to raw evidence.

Figure 3.

A concept sketch of an annotated GeoTime scene representing the events of a fictional story.

Full figure and legend (240K)

The assembly and organization of story elements is also important. A story provides a unique representation of events that are not necessarily linear. In timeline visualizations, events are ordered chronologically. In a story, however, events can be arranged in an order relevant to the topic. The author of a story can move the reader's attention forward and backward in time to provide context. We refer to this important concept of sequencing the presentation of information as chronotopic time.

The three key workflow objectives for the prototype were to

• Integrate new and existing pattern matching tools to drive annotated expressive visualizations. This should assist the process of story discovery.
• In a narrative text, provide links to supporting visualizations of events in time and space that invoke a complete analysis of context and allow jumping off points for further analysis. This assists with story authoring and storytelling.
• Support authoring and comparison of multiple versions of story sub-components and elements. This helps make visible alternative sequences, explanations, and hypotheses.

System implementation

The GeoTime Stories prototype includes three main components: annotated pattern detection, narrative text authoring, and story threading. Each of these is designed to provide particular analytical capabilities and is discussed individually.

Pattern detection and expressive annotation

A common problem for analysts is that they must wade through large volumes of low-level data and painstakingly extract behaviors and patterns in order to understand a situation. This process reduces both the amount of data that can be analyzed and the number of observations that can be discovered. To address this, the stories system provides functions for detection of simple patterns in movement activity, including discovery of possible meetings, speed of movement, percentage of time spent at unique locations, and identifying gaps in observations. The results of these functions are then fed into the annotation system in order to reduce the cognitive effort required to recognize these low-level incidents in the data. This allows the analyst to work at a higher level of abstraction, thereby stimulating faster recognition of larger behaviors or plots.¹⁹

It was not our focus to develop new pattern discovery functions; rather, it was our intent to understand how they could be leveraged into the analytic workflow for maximum benefit. Although pattern detection is a separate area of research of great complexity, the pattern functions we implemented were useful in testing and refining interaction between pattern detection, visualization, and a story. The pattern system uses a plug-in architecture so that additional external pattern detection functions can be integrated.

GeoTime makes use of visual annotations to highlight the results of pattern searches within the visualization display. Annotations are graphic cues that can be added to a space–time scene in order to mark and describe important events in a given view. An individual pattern function applies specific annotations to the data points identified with an instance of a pattern. For example, in Figure 4, the movement speed pattern function annotates a geo-temporal path with a visual indicator of velocity. It can also annotate this with icons that express speed by indicating a mode of transportation such as moving by foot, car, or aircraft as shown in Figure 5.

Figure 4.

Annotated snapshot result of the velocity annotation function. Path of movement is annotated with indicator of speed; red is fast and white is slow.

Full figure and legend (295K)

Figure 5.

Annotation of movement speed using icons (cars and walking people) to describe possible mode of transport.

Full figure and legend (268K)

Annotations may also be manually applied to events by an analyst to highlight events and direct the attention of the story 'reader'. The analyst selects events in the scene and then chooses from a palette of annotations, including curves, arrows, callouts, outlines, and enlargement, shown in Figure 6.

Figure 6.

The user annotation menu with five annotation styles.

Full figure and legend (315K)

GeoTime annotations are unique in that they are implemented as data objects referenced to the actual event data, rather than as a separate graphical overlay. This approach is required due to the dynamic nature of the GeoTime 3D display. In order to remain attached to the events they reference, annotations need to be re-rendered each time the scene updates in response to user interactions. Although this adds complexity to the software, a significant benefit to this approach is that annotations retain a semantic connection to their events and patterns. It also allows for richer graphical and data-specific expressions, such as enlargement or re-coloring of events, and even supports interactions such as mouseovers and menus on annotation graphics themselves. One possible downside is that annotation options are limited and not as simple as 2D sketch interfaces.

Annotations are implemented such that their display is tied to a saved 'snapshot', which is an embedded link in the text of the story window. The story window is the authoring environment where the visualization of events in space and time, including annotations, are meaningfully connected to a story.

Story window

The story window, as shown in Figure 7, is the focus of story-authoring functionality in GeoTime. It enables the analyst to review pattern search results, author and present stories about events in time and space, capture moments of insight, and collaborate with other analysts.

Figure 7.

The story window to the right of the GeoTime scene. Text sections titled 'Pattern: ...' are generated by pattern functions.

Full figure and legend (387K)

A story document is comprised of a title and one or more text sections. Each text section can be any length or number of paragraphs and has an optional title bar. Story templates are outlines that can be selected when a new document is created. For example, a template may contain empty sections titled 'Introduction', 'Body', and 'Conclusion', or may include detailed instructions for how to do certain types of analysis. Story templates are meant to jumpstart analysis or provide instruction to new users about tools and techniques. The content of the templates is configurable. GeoTime currently contains very simple templates as examples.

The story window is designed to address a top-down story-based workflow by allowing the analyst to capture their insights, thoughts, explanations, and conclusions about events visualized in GeoTime. The tool provides the analyst with a familiar text editor interface for assembling a story but adds the capability to embed clickable links to captured GeoTime visualization states, called snapshots. A distinction should be drawn here between this implementation and common hypertext systems. In this case, the link affects the display of the GeoTime scene and does not change context or content in the text window where the click occurs. Also, the link invokes a view of events in space and time, which are the basic elements of a story that one is reading. A suitable analogy of what is possible is that of a person reading a novel who can instantly invoke corresponding scenes from a movie version of the novel on a nearby screen.

Envisioning this in the context of GeoTime analysis, the analyst is therefore able to save a view of a time–space region containing events that support their stated analytical claims. Similarly, GeoTime also allows them to annotate and describe ideas about specific events. For example, one could use the text of the story 'Little Red Riding Hood' and include annotated snapshots of exactly where in the forest the Wolf and Red Riding Hood's space–time paths meet (assuming GPS tracking data were available). Digging deeper, one could then point out if they just happened to cross paths or if the wolf was waiting for someone to walk by (this distinction would be apparent in GeoTime).

Snapshots are essentially references to a display state in GeoTime. They capture all the required parameters to reproduce a view, including geographic extents, time range, annotations, point-of-view, and visibility of each data element. Snapshots are created by clicking on the 'take snapshot' button, which then saves a snapshot and embeds a link to it in the text at the cursor. Each snapshot is uniquely named and placed in the preview list below the text editor, which contains thumbnail images of all snapshots in the current document. Snapshots can be dragged from the preview list into the text body to create a new link or one can simply type the name of a snapshot in the text and it will be automatically detected and highlighted. Links use a common syntax – numerals inside square brackets. Snapshots can be placed anywhere in the text any number of times. A preview image of a snapshot pops up when mousing over a text link.

When a snapshot link is invoked, GeoTime reproduces the exact view state when it was created. This is not a static view; the user is able to adjust any parameters or continue analyzing as usual from this point on. This allows the reader to further examine the exact data that the author has referenced, using the same tools that were used during authoring. For example, one could zoom out to see more context or animate the events over time. One can also add additional annotations and analytical depth and then feed this back into the story as a new snapshot or text section. Existing snapshots can be updated in the preview panel.

It is worth noting that two different temporal systems are linked together through the snapshots. GeoTime presents events in chronological order and the nature of text provides the ability to arrange the sequence of information in any order – part of the art of storytelling. For instance, an analyst may wish to insert a reference to events that occurred 20 years prior to the rest of the story. Using story text and snapshots, this sequence can be laid out for a reader to read and click through. If only the GeoTime view existed, the reader would have to scroll through the data linearly in time and be left to guess at the relevance of the earlier events. In some cases, a linear view of the events in a story may be desirable as an alternate and objective perspective to the text. This is accomplished by simply turning attention to the GeoTime visualization and scrolling through events using the time slider. One feature for future exploration is to show snapshot boundaries in time and space and then provide links back to relevant story text, thereby closing the loop between the two views.

The story window is also leveraged by the pattern matching system for displaying results to the user. Each time a pattern search is executed against the data set, a new text section is generated containing a textual description of the nature of the pattern and a summarization of the results. Links to snapshots of the pattern instances in time and space that can be clicked for rapid review and/or confirmation by the analyst are also included. In Figure 7, the text section titled 'Pattern: Meeting Result' is an example of this. The text and links can be incorporated anywhere in the text using standard cut and paste actions.

Aside from the formal application of the story window in developing a story, the system can also be used to aid in unstructured free-flowing thinking. The story window and snapshot system can be used to generate view mementos or bread crumbs of an analyst's progression through information. In this sense, the story window serves as a notepad and provides a convenient location to assemble thoughts and threads of exploration. Some users have used one text section as a scratch pad while authoring a final report in another text section.

Although elements of the story window are familiar, the underlying implementations and focus on storytelling for events in time and space make it uniquely suited to many analytical tasks. Similarity with standard interfaces helps to improve trainability and potential adoption by analysts. The ability of the story window to organize visualizations in a user-centered unstructured or structured narrative space is a concept that has potential to extend to other visualization systems.

Story comparison and collaboration

An important analysis technique is to evaluate alternative explanations that may account for some given evidence.¹⁷ The ability to enumerate multiple possible explanations within a story is therefore desirable. Further, multiple analysts may be involved in contributing to the analysis or providing feedback about a story. The story window supports these activities by providing a simple system for categorizing and filtering sections of story text.

Every text section includes a color bar on the left side and the colors can be changed with a right-click option. Figure 7 illustrates how this appears in the interface. This mechanism allows the analyst to define categories within the text body. The meaning of the colors is not pre-defined and the analyst can define this based on task. For instance, in the context of 'red team' analysis (i.e., the devil's advocate point-of-view), a useful application of color would be to mark core sections of a report blue and use red sections for red team input. If the use case was to indicate the correctness of another analyst's work, red may be used to indicate a problematic section. Feedback, role-playing, versioning, change tracking, brainstorming, war gaming, details, back story, alternatives, and comments can also be indicated.

Immediately below the story window is a row of filter buttons that allow the user to toggle the display of text sections by color. By switching content on and off, the analyst is able to include or exclude certain threads, details, or opinions in a story, thereby changing its possible readings. One application would be to color code one version of the story blue, and a competing version of the story green. Common elements could be yet another color. The filters would allow the analyst to quickly flip from one version to another to compare explanations for the same set of observations. Story branches can also be set up by providing instructions within the text to turn certain section on and off at key points. For example, one could write in the text 'click green sections ON if you agree or red sections ON if you disagree.'

Flexibility and simplicity for the analyst were favored over task specialization in the design. We believe there are many ways to use text sections that we have not considered. Further investigation is necessary to determine utility and scalability. It is our intent that an analyst or organization will develop techniques and protocols that suit their needs.

Technical architecture

The stories system consists of four primary components: the visualization, the pattern matching system, the story window and the capture management system, shown in Figure 8. The capture management system acts as a hub for view storage and routing invocations to visualizations.

Figure 8.

The technical architecture of the stories system.

Full figure and legend (70K)

The stories system is written as a Java plug-in for GeoTime. Although the implementation described here is specific to GeoTime, the architecture is flexible enough to integrate with other visualizations provided they can store and recall snapshots.

Capture management system

The capture management system stores and invokes snapshots of visualizations – in this case GeoTime space–time views. It uses the inversion of control design pattern. A visualization must provide the ability to capture its current view upon request. This snapshot is stored along with an image of the snapshot scene and is given a unique reference. Invoking the reference at a later date will direct the visualization to recall the exact view state. It is our assumption that any visualization that implements a capture system can be leveraged as supporting evidence within a story. For example, a story about a corporate scandal could include snapshots linked to organization charts or stock price visualizations. Recent work^{12, 13} demonstrates this use case with charting visualizations utilized in discussions.

Story window

The story window is essentially an HTML text editor that supports embedded references as hypertext links. The system monitors a list of available named snapshots in the document and highlights the names if they appear in the text. Clicking on one of these links invokes the capture manager to fetch and display the stored snapshot in the visualization. The story window contains additional functionality to display multiple html viewers (the text sections) by a color-defined category. This category is used by the filter interface to hide or show text sections.

Visualization and annotation system

GeoTime was used as the visualization system in this research because of its applicability for representing stories about events in time and space. The application is capable of capturing the complete state of the visualization, including geospatial and temporal bounds, viewpoint, and visible data. These can be recalled by the capture management system.

GeoTime's interface and rendering system was extended to include the annotation types described previously. These can be captured and recalled by the snapshot system Annotations can be activated by external functions, as in the case of patterns, or can be manually added by the analyst.

Pattern matching system

The pattern matching system interacts with the GeoTime data system, the story window, the capture management system, and the annotation system in order to run searches and display results. The pattern matching system includes an API to allow third-party developers to write patterns that scan over the GeoTime data. The function must set up annotations and add them to the data and then output a text description of the results, including snapshot descriptions, to the story window.

Informal evaluation

Over a period of 6 months, the Stories capabilities were reviewed with intelligence analysts for feedback and the pattern annotations system was tested against both reference data sets and actual operational data. Test data included GPS vehicle tracks, personal transaction billing data, including phone calls and credit card transactions, and proprietary operational data sets of varying size and nature. In order to support these different data types, the pattern system was enhanced with controls to adjust for data precision, frequency, and scale.

Procedure

In all cases, the test data sets were loaded into GeoTime and presented to analysts by the researchers. The pattern search and annotation functions were executed, generating results in the story window, then reviewed by clicking the links provided in the text to zoom and highlight instances of patterns in the GeoTime view. Examples of stories based on the sample data were presented.

Feedback

In all cases, feedback was enthusiastic. It was noted that the methods used by GeoTime Stories to display pattern search results are well suited to support rapid review by an analyst. In some cases, analysts were working on their own pattern finding functions, but did not have a way to present results effectively. Other feedback referred to the story-telling capabilities and how they could be used in analysis tradecraft training and for reporting discovery to peers and decision makers. Feedback quotes include:

• 'The annotated story is something that could be presented to high-level decision makers. It is obvious to see what is happening without having to know details of the data.'
• 'The potential for using GeoTime Stories to capture and communicate tradecraft processes for both reporting and training is significant.'
• 'Stories would be good for explaining tradecraft. Each type of data and situation is different, with its own properties and patterns. You need to be able to look at it – turn it around and understand and explain to someone new: 'These are the important things you will see in data – you might think this artifact looks like behavior X, but in fact it might be something else...' This can be explained with examples using Stories.'
• 'We need to be able to capture and communicate the analytical process. Stories let us follow a sequence of steps towards a solution but also go back and follow-up on branches that were missed.'

Analysts had several informal recommendations for improving the prototype system, including

• Expand the number of different graphical options for manual annotation of the event data.
• Hide annotated events so that only the annotation is displayed, in order to simplify the display.
• Enhance the story system to support snapshots to other visualization systems was also an expressed desire.
• Provide a way to tag and add more categories of story text sections is necessary.
• Export story documents as standard Microsoft-compatible formats.

Additional pattern functions and tuneable pattern parameters were deemed necessary. Specific proprietary pattern functions were of interest. Additional work would be required to integrate them with the Stories pattern API.

Findings

Our findings mostly pertain to the utility of the pattern system in finding and expressing patterns reflecting the bottom-up aspect of the GeoTime Stories capabilities. For almost every data set, it was necessary to adjust pattern-finding parameters to manage the relevancy of results. This was expected due to differences in geospatial scale, precision, frequency, and other characteristics of the data. For example, a meeting may be defined as two people located within 50 min and 5 min of each other, or, at a global level, it might be two people in the same city on the same day. The utility of a pattern algorithm will depend on many factors, including the character of the data and the analytical task. It was beyond the scope of this research to automatically tune algorithms for any given data set; however we believe such a capability is necessary to provide a truly automated pattern annotation system.

In the sample data we tested, the meeting finder successfully identified potential meeting times and locations. Analysts liked being able to step through snapshots of each meeting result set in the text window. Feedback on the movement speed annotation function was that it provided a quick sense of the behavior of a track. Annotating extreme-speed movements was considered especially useful because it could indicate errors in the data and other conditions of special analytical value.

There was expressed interest and encouragement to continue the work and pursue ongoing tests and development in operational environments. Analysts noted, 'You are going in the right direction with Stories in all three areas – patterns, annotation, and narrative. We could use this now.'

Formal evaluation and development

In order to more fully assess the new capabilities and usability of the Stories tools, we conducted a preliminary study in conjunction with NIST. The evaluation collected analyst-centered data focused on usability and utility. The study was designed to specifically test how well the analyst could integrate the narrative text, snapshots, and pattern results into a text report; and how well these reports could be understood by others who are unfamiliar with the situation or data. The narrative tool, patterns, and annotation system were evaluated individually for their utility to support asynchronous collaboration and the usability of each component. The evaluation was carried out by NIST over a 2-day period with three participants with experience in intelligence analysis.

Procedure

Three intelligence analysts with experience in geospatial analysis were selected by NIST for this evaluation. The analysts identified geospatial intelligence analysis as their primary workplace responsibility, had a range of 5–29 years working with geospatial information and had self-identified as experts in this field.

Participants were given training on the first day to establish a basic understanding of GeoTime functionality and usage as well as an introduction to the story functionality. The participants were evaluated for competency with GeoTime functionality at the end of the day.

For the evaluation, participants were presented with a fictional scenario designed to mimic ones that might be encountered in a criminal investigation. The exact analytical task description was incorporated into the provided scenario with the Stories window (see Figure 9). Participants were restricted from using Microsoft Word, Microsoft PowerPoint, or any other note taking tool.

Figure 9.

An analyst authored GeoTime Stories report produced during the NIST evaluation. Images are views of the user-created snapshots corresponding to the blue hyperlink number in the text body. The 'Background' and 'Task' titled sections were provided to the analyst as a starting point. The first analyst wrote the green text sections, and a second analyst added the orange 'Comments' text section after reviewing the first analysts' report. We believe the second analyst also wrote the 'editor's notes' throughout. (Note: Imagery provided by DigitalGlobe)

Full figure and legend (527K)

The evaluation was divided into two stages. In stage 1, the participants were given 3 h and were asked to write a report analyzing the information in the scenario. It was expected that the analysts use the Stories authoring, pattern, and annotation tools to achieve this goal. In addition, they were told that their report would be handed off to another analyst at a later period. In the second stage, the analysts were presented with a new scenario they had not previously seen, and an associated report authored by another analyst. The participants were not informed that this work was from another participant. They were asked to review the report, including clicking on saved views, etc., and add feedback, comments, or interpretation to the report using the provided tools for an additional 2 h.

Results

The results were collected, evaluated, and summarized by NIST to avoid experimenter bias. Their findings are summarized below.

NIST observed that the participants were in control of their interactions with GeoTime Stories throughout the experiment. It was observed that GeoTime Stories did not impose workflow on the participants, and empowered them to create work flows of their own choice, which the participants determined to be a strength.

Participants gave high praise to GeoTime Stories' ability to support asynchronous collaboration. The participants were successfully able to build upon other participants work in the evaluation. The average rating over five dimensions of continuing someone else's work was 6.4 out of a possible 7.0. Participants expressed the highest confidence, 7.0, that their reports could be understood by others.

The participants gave high marks for the functionality and usefulness of the snapshots awarding 6.3 and 7.0, respectively. Participants also gave high marks for the annotations, giving 6.0 for usability and 6.33 for usefulness. The participants expressed that they wanted more versatility in the annotations; specifically they wanted the ability to annotate more things in the view, and wanted more power to edit the annotations.

Participants gave high ratings of 6.33 for both usability and usefulness for the text components. The participants requested more editing features and more control over changing the colors associated with text sections.

Participants reported that it was easy to learn to use individual aspects of GeoTime Stories (e.g. rating the learnability of annotating 6.33; creating callouts 6.67; taking snapshots 6.67; manipulating text sections 6.00; executing ground speed pattern 6.67; executing meeting pattern 6.67; authoring report 6.67). When evaluating the system as a whole it received an average rating of 5.33. The participants stated their concern that consumers of the report would require GeoTime software and training in order to view the report.

In addition, the NIST team generated numerous excellent suggestions for changes to the Stories prototype to improve user experience and effectiveness.

Sample analyst report in GeoTime stories

Figure 9 shows a report authored in GeoTime Stories by experienced analysts during the NIST user evaluation. The main text body is shown, along with images of some of the snapshots included in the report. It incorporates different sections of text containing the task description (gray), the analysts report with summary and detailed analysis (green), and a second analysts input (orange). The report includes links to snapshots of GeoTime views containing user-defined annotations.

Although we did not perform a formal analysis of the report content, we found them to be very revealing. Perhaps most interesting to us was how effectively the authored text can become the voice of the analyst when precisely connected to a visualization of the events they are describing. This is a subjective observation, but seems important enough to justify further investigation of the qualitative aspects of this style of presentation.

Analysts verbalized insights and thoughts in different styles; both carefully measured language and immediate informal notes coexist in the sample report. The hypotheses and interpretation scattered throughout the text, provide insight into how an analyst interprets specific evidence. The second analysts' feedback to the first reveals how different assumptions, opinions, and personalities can contribute to analysis. This leads to questions about how capturing emotion and other qualitative properties such as expression of uncertainty or skepticism might improve the communicative power and accuracy of analytical reporting, and further, how tools might be designed to include or encourage different styles of discourse. Perhaps this could even be adjusted to be appropriate to the task, or an organization's policy.

Also of note is the precise language and the exact times and locations included in the report, despite the presence of connected reference data and tools that provide access to these details. This is likely a habit or requirement resulting from use of tools that do not allow drilldown into linked, supporting evidence.

Conclusion

Storytelling is an effective means of communication and collaboration and is used by analysts for reporting complex events to other analysts and decision makers. In this paper, we have investigated how the concept of story relates to the process of analysis and how it can be supported in the GeoTime visualization software. A model of story concept was developed and two approaches were identified to guide the design of the system:

• A bottom-up approach, in which patterns in a visualization of low-level data are annotated so that analysts can quickly recognize higher level behaviors using their tacit knowledge.
• A top-down approach, in which the analyst can preserve moments of insight and explain them to others in the form of a story that can be told to, or read by others.

We have implemented mechanisms to support these approaches. First, to support a bottom-up approach, the pattern system automatically detects basic pattern types in event data and then graphically annotates them within GeoTime. To support a top-down approach, the story window allows for authoring and telling of analytic stories and provides the means to connect explanation to views of events in time and space. Thus, the analyst can link analytical thought and hypotheses to moments in information time and space. We have also added the ability to categorize parts of a story and toggle their display as a way to insert detail or collaborate on content. We believe this mechanism also enables the representation of multiple versions of stories and perspectives.

It is our intent that these capabilities will help analysts communicate insights and explanations in an expressive analytical product and that the ability to perform analyses at a higher level of abstraction – the level of a story – is made possible. Feedback from reviews with analysts was positive and encouraging and indicates that these capabilities could reduce the time analysts spend on low-level data analysis and report generation. In addition, a NIST-led evaluation of GeoTime Stories showed that these capabilities received high marks for usability and utility from experienced intelligence analysts.

We believe that these techniques can be applied in other visualization systems and that visualizations of data are more effective if they are linked to explanations. Ideally, all visualization systems should include a scene-capture mechanism that can be integrated with external systems.

Future work

Our next steps will include additional testing with analysts and identification of selected story detection, storytelling, and sense-making capabilities to be included within future versions of the deployed GeoTime analysis product. This should improve GeoTime by reducing visual clutter, increasing scalability, reducing effort and, most importantly, improve the interpretation and comprehension of significant patterns within the data.

We also anticipate that GeoTime Stories is an effective way to capture and teach tradecraft to new analysts. The ability to incorporate explanation of tradecraft within an actual analytical tool is a significant benefit. This could be superior to classroom training, manuals, on-line training, and other common techniques. It seems that GeoTime Stories could provide a one-on-one with expert training experience in a repeatable and flexible form.

References

1. Fisher WR. Narration as human communication paradigm: the case of public moral argument. Communication Monographs 2004; 51: 1–22.
2. Herman D. Spatial cognition in natural-language narratives. Proceedings of the AAAI Fall Symposium on Narrative Intelligence (Cape Code, North Falmouth, MA, USA), AAAI: Menlo Park, CA, USA, 1999.
3. Harper R, Kapler T, Wright W. Correlating events with tracked movements in time and space. In Proceedings International Conference on Intelligence Analysis (McLean, VA, USA), MITRE: Washington, DC, USA, 2005.
4. Plowman L, Luckin R. Designing multimedia for learning: narrative guidance and narrative construction. CHI'99: Proceedings of the SIGCHI conference on Human factors in computing systems (Pittsburgh, PA, USA), ACM: New York, NY, USA, 1999; 310–317.
5. Laurel B. Computers as Theatre. Addison-Wesley: New York, USA 1993.
6. Vaucelle C. A system to compose movies for cross-cultural storytelling: textable movie, Proceedings of TIDSE (Darmstadt, Germany), Springer: Berlin, Germany, June 24–26, Vol. 3105, 2004; 126–131.
7. Joshi D, Wang JZ, Li J. The story picturing engine–a system for automatic text illustration. ACM Transactions on Multimedia Computing, Communications and Applications 2006; 2(1): 68–89. | Article |
8. Zellweger PT, Mangen P, Newman P, Reading and writing fluid hypertext narratives. Proceedings of the thirteenth ACM conference on hypertext and hypermedia (Washington, DC), ACM: New York, NY, USA, Conference on Hypertext and Hypermedia, 2002; 45–54.
9. Plaisant C, Milash B, Rose A, Widoff S, Shneiderman B. LifeLines: visualizing personal histories. In: Proceedings of CHI '96. (Vancouver, BC, Canada), ACM: New York, 1996; pp. 221–227.
10. Gemmell J, Bell G, Lueder R, Drucker S, Wong C. MyLifeBits: fulfilling the memex vision. ACM Multimedia '02, December 1–6, 2002, Juan-les-Pins, France, ACM: New York, 2002; 235–238.
11. Gee K. The ergonomics of hypertext narrative: usability testing as a tool for evaluation and redesign. ACM Journal of Computer Documentation 2001; 25: 3–16. | Article |
12. Heer J, Viegas FB, Wattenberg M. Voyagers and voyeurs: supporting asynchronous collaborative information visualization. CHI 2007 Proceedings (San Jose, CA, USA), ACM: New York, NY, USA.
13. Website www.Many-eyes.com, downloaded 18 July, 2007.
14. Gershon N, Page W. What storytelling can do for information visualization. Communications of the ACM 2001; 44(8): 31–37. | Article | ISI |
15. Kapler T, Wright W. GeoTime information visualization, IEEE InfoViz (Austin, Texas, USA), Palgrave Macmillan: Houndmills, Basingstoke, Hampshire, England, 2004.
16. Kapler T, Wright W. GeoTime information visualization. Information Visualization Journal 2005; 4(2): 136–146. | Article |
17. Heuer R. Psychology of Intelligence Analysis. Center for the Study of Intelligence, Washington, DC, USA, 1999.
18. Johnston R. Analytic Culture in the U.S. Intelligence Community. Center for the Study of Intelligence, Government Printing Office: Pittsburgh, PA, 2005.
19. Xiao L, Gerth J, Hanrahan P. Enhancing visual analysis of network traffic using a knowledge representation. IEEE VAST 2006 (Baltimore, MD, USA), IEEE: Los Alamitos, CA, USA, 2006.