Introduction

Visual analytics is defined as the science of analytical reasoning facilitated by interactive visual interfaces.¹ Mobile visual analytics extends the visual analytics process using state-of-the-art mobile devices to increase the effectiveness and interactivity of on-site analysis. These types of solutions can provide advanced analytical insight to first responders and public safety command personnel by allowing them to analyze and understand on-scene, active emergency situations through interactive, integrated data analysis, and visualization.

Such mobile analytics solutions can benefit first responders in a variety of ways. First, the mobility of the handheld devices using wireless connectivity can minimize the 'fog of war' effect allowing first responders to better carry out rapid and actionable on-site decision making. Furthermore, these tools can provide improved situational awareness and support first responders in planning immediate life-saving responses and prioritizing actions in emergency situations. Second, the rapidly growing capabilities of mobile devices (e.g., PDAs and cell phones) provide a ubiquitous environment for deployment within a variety of fields. However, most mobile devices still have many limitations including small screens, limited user interfaces, a short battery life, low bandwidth of the system bus, slow CPU clock speed, limited storage capacity, and a lack of advanced graphics hardware.

Our goal is to make mobile devices valuable tools for emergency response by effectively visualizing relevant, selected information (e.g., images, videos, 3D models, and sensor data streams) on devices with varying capabilities and resolutions. To this end, we have developed a mobile visual analytics system that processes and displays sensor, location, and video data for first responders to enhance situational awareness and enable more effective decision making. Previously, we introduced a prototype visual analytics system for emergency response on mobile devices.² In this paper, we extend our previous work and present an enhanced visual analytics system for emergency response and training with congestion visualization, video playback, and reinforced visualization of 3D personnel and scene models.

This paper is organized as follows: The following sections discusses the background and then summarize visual analytics for emergency response. The further sections describes the design of our system and presents visualization and analytics on client mobile devices. The next further section gives a brief summary of implementation and results of our system. The penultimate section discusses the capabilities and potential of our system as a visual analytic tool for emergency response. The final section presents conclusions and discusses some possible extensions for visual analytics.

Background

The related work is classified into three categories: visualization of sensor data, visualization on mobile devices, and visual analytics on mobile devices.

Visualization of sensor data

With the increase in applications for sensor networks, manipulation and visualization of sensor data streams have become crucial components in the effective analysis of network data. Fan and Biagioni³ described approaches to process and interpret data gathered by sensor networks for geographic information systems. These approaches combine database management technology, geographic information systems, web development technology, and human computer interactive design to visualize the data gathered by wireless sensor networks. Koo et al.⁴ designed software to analyze multi-sensor data for pipeline inspection of gas transmission. The information gathered by sensors is parsed and converted before it is saved in a database. Gunnarsson et al.⁵ introduced a mobile augmented reality prototype for visual inspection of hidden structures on mobile devices using data collected from a mobile wireless sensor network. Pattath et al.⁶ implemented an interactive visual analytic system to visualize sensor network data during football games on PDAs.

Visualization on mobile devices

Many researchers have examined ways to effectively display complex 3D models on mobile devices. We can divide this research into two categories: the simplification of representations and the effective transmission of data. Simplification of representations^{7, 8, 9, 10, 11} makes it possible to visualize complex 3D models with a limited graphical capability while effective transmission^{12, 13} between a server and a client enables mobile devices to visualize more complex models.

Hekmatzada et al.⁷ described non-photorealistic rendering of 3D models based on a server and client environment. Their work provided transmission of meshes from a server progressively, as well as level of detail (LOD) rendering, allowing clients to navigate through the data in nearly real-time. Diepstraten et al.⁹ proposed a remote line rendering technique between the server and client. The server extracts feature lines of 3D models and transmits them. The clients then draw the transmitted results. Thus, the clients do not need to have high computational capabilities since they only draw 2D lines. Duguet et al.¹⁰ displayed complex geometry on mobile devices by using a point-based rendering technique. Quillet et al.¹¹ presented two optimization methods to visualize an urban environment on mobile devices interactively. One optimization method extracts feature lines and then changes the lines into vector lines. The other splits the urban environment into cells in order to transmit them as a stream. Their work also provides an efficient LOD solution. Pouderoux and Marvie¹² proposed two levels of adaptivity to display a large amount of terrain data regardless of the device. The terrain data are partitioned into regular tiles and the tiles around a viewer are transmitted as a stream. The tiles are rendered using a pre-computed triangle strip path. Zhou et al.¹³ introduced a client-oblivious framework, which integrates volumes and iso-surfaces into one hierarchical structure to visualize volumes on mobile devices.

While much work on visualization on mobile devices has focused on 3D rendering, 2D graphics and visualization can be just as efficient in the case of information visualization. In fact, there are many applications that utilize 2D capabilities of mobile devices in fields such as geographic information systems,^{14, 15} entertainment, education, business, and industry. Moreover, OpenVG¹⁶ and Mobile 2D Graphics (M2G) are boosting the development of more 2D applications that are scalable across any screen size.

Visual analytics on mobile devices

The application of visual analytics to mobile devices has several challenges. It is different from visual analytics on common desktop systems because of the restricted display space and computing resources of mobile devices. Sanfilipa et al.¹⁷ introduced InfoStar,¹⁸ an adaptive visual analytics platform for mobile devices. Their work was demonstrated at Super Computing 2004 (SC2004) providing information such as maps, schedules, exhibitor lists, and visual exploration tools to conference attendees. Similarly, the work by Pattath et al.⁶ provided a visual analytic tool for the visualization of network and sensor data gathered from Purdue Ross-Ade Stadium during football games. Finally, Herrero et al.¹⁹ presented an intrusion detection system for visual analysis of high-volume network traffic data streams on mobile devices.

Visual analytics for emergency response

For emergency response, a well-designed visual analytics system is necessary. The display capability must be tailored to the responders and their roles, and provide a succinct, quickly understood display of relevant information extracted from all information acquired. For example, Special Weapons And Tactics (SWAT) teams are highly trained groups of police officers whose missions include hostage rescue, dignitary protection, and high-risk warrant services. These missions all require successful coordinated information collection and exchange. In the case of the SWAT team responding to an active shooter in a school, the first and most critical requirement of the team leader, as well as all responders, is to have the most accurate sense of situational awareness as possible. In order to do this, the responder must answer the following questions:

• Where are all team members located?
• Where are the locations of responding personnel?
• Where are the secure, neutral, and hot zones of the incident?
• What locations provide opportunity or threat information?

The goal of our system is to efficiently provide answers to these questions in order to enhance situational awareness.

In addition, the capability to provide information back to the emergency operation center, such as indicating rooms cleared or information contradictory to current situational assessment, is also vital. As previously indicated, relevant information is specialty-dependent. A firefighter responding to a fire at the same building would need some of the above information, as well as task-specific information, such as fire spread, potential toxic gases, or locations of dangerous goods stored.

Moreover, the first generation of mobile analytics should target readily available technology, such as PDAs and smartphones. This display system will be useful in decision support during emergency response, as well as planning for event response. The system will allow responders to reduce the time spent on information gathering and instead focus on response actions, such as asset dispersal. These actions will be assisted with a visual display of current information while after-action reviews (AAR) will also be enhanced by providing users with the ability to better see potentially unknown information such as evacuation routes that are not used efficiently. After-action review methods are also aided through real-time analysis of actions taken during response time. This system will enhance training for response to many unique emergency situations, and we have deployed a version of this system at our in-field emergency response training site. The results seen here show its application to both simulation data and previously recorded first-responder training exercises.

System design

Figure 1 shows the abstraction of our system structure as a server–client architecture. Our system focuses on the utilization of various types of data sets such as images/videos, 3D models, sensor data, and text data. All of the streaming data are received from and preprocessed by each server in the server group. In Figure 1, the data converter in a server group converts all input data into the appropriate types for the mobile visual analytic client.

Figure 1.

System overview.

Full figure and legend (115K)

This conversion is necessary for visualization on mobile devices and involves determining the appropriate representation of the data for rapid, in-field cognition on a small-screen mobile device. The data created for desktop systems cannot be used for mobile devices without further preprocessing because of the limits of mobile devices in terms of memory, bandwidth, and screen resolution.

The converter preprocessing has components that are customized for each type of input data stream, but in general it uses a flexible structure to allow the input of a variety of data based on the given response situation. Moreover, the structure is designed to allow tailored processing of the same input data for different response situations and different roles in the response.

As our previous work,² we are using pre-generated/recorded data so that all of the information can be initially transmitted to the client. However, our system could be modified to get real-time feeds.

Our mobile visual analytics tool consists of a 2D/3D visualization component that shows personnel-related information, situational and static scene related information, integrated multimedia playback functionality for personnel outfitted with cameras, and fast-forward/rewind capability for reviewing events.

Input data

We categorize the input data into three types in terms of their functionalities. One is personnel-related information for moving entities. Each moving entity has identification, position, and time stamp information. Second is situational information. Examples of this include temperature, dispersion of toxic gases, and water contamination. The third is static scene-related information such as a 2D map or a 3D model that is used for representing the environment.

Preprocessing in a data converter

In our current system, input datasets can also be roughly categorized by their real-time properties. The abstracted/simplified background images as well as the 3D models are immutable data during processing for visual analytics, whereas the other data sources (e.g., sensor/video data, location data) are time varying. Therefore, our data converters are designed to process both time varying data and static data.

For the static data (e.g., images/blueprints, 3D models, text files), conversion to a format that enables execution on a mobile device in real-time under the management of the server is performed. Using vector images gives our system scalability across screen size and resolution. Hence, all images are converted into vector path data in scalable vector graphics format.²⁰

For the conversion of time-varying data (e.g., sensor/video data, location data), special processing is needed to provide proper synchronization of the temporal data streams in a networked environment. Moreover, filtering and selection of large data streams is necessary to enable real-time use on processor- and memory-limited mobile devices.

To effectively utilize large streaming data on mobile devices, we employ simple compression/transformations of the data to reduce both network bandwidth and local storage requirements. We also use data importance characteristics to determine update rates, data items to skip, and interpolation methods to maintain our performance requirements. Finally, level of detail, level of abstraction, and level of data aggregation are chosen not only to enable interactive performance but also to reduce visual clutter and enable effective visualization and analysis on the mobile device.

Management of streaming data on the client

To deal with the large size of time-varying data streams, we need to utilize an appropriate data structure for storage. We use a circular queue, as shown in Figure 2, in order to minimize memory consumption. A circular queue is a particular implementation of a queue in which insertion and deletion are totally independent. Although our system focuses on client-based visualization and analytics, such queuing structures can be used for processing of streaming data in server–client architectures. In our work, the size of the circular queue is set to 30 to provide short-term reference data for visualization and still fit in the memory of PocketPC phones and PDAs. In Figure 2, the simulation data manager serves as a communication handler in a server–client system and can be composed of several types of data managers. The data manager is responsible for updating the appropriate entry type in each element of the queue (e.g., sensor, location). The application in a main thread can then take the data from the queue by time stamp without suspension occurring due to any network traffic.

Figure 2.

Data structure for streaming data.

Full figure and legend (103K)

Visualization of data

In our system, we consider several visualization issues to represent various data types effectively. Dynamic and static categories can be applied to moving entities and stationary scenes, respectively. Temporal and spatial categories describe information changed at each time and position. Aggregation and non-aggregation categories highlight personnel-related information in terms of a group or specific individual information.

From the point of view of input data for the visualization of personnel-related information, we show the position, path traveled, health/activity level, evacuation status, congestion, area traveled, and corresponding video data. For situational information, our system visualizes sensor data, examples of which are the distribution of temperature and toxic gases. In addition, our system provides a perspective view of the 3D environment. These functions can be utilized for situational awareness and assessment.

Visualization on mobile devices

The main concerns for visual analytics on mobile devices are the device limitations (screen size, memory) and the appropriate data aggregation/abstraction level to enable effective decision making. In our work, the device memory limitations are primarily solved on the server component during data conversion to an appropriate representation. Hence, our mobile visualization client mainly deals with the visual representation to enable visual analytics on mobile devices for emergency response and training.

Case studies

To demonstrate our system capabilities we have employed two case studies: a simulated evacuation of The Station nightclub fire that occurred in Rhode Island in 2003 and a testing exercise for a rescue operation in an elementary school.

Case 1: The first scenario has been completed from an investigation and a computer simulation by the National Institute of Standards and Technology (NIST)²¹ after the fire. We used two simulated data sets including fire data and evacuation data of 419 personnel. The fire was simulated by the Fire Dynamics Simulator (FDS4)²² developed by NIST. FDS4 is a computational fluid dynamics model of fire-driven fluid flow and provides time-resolved temperature, carbon monoxide, carbon dioxide, and soot distribution throughout the building for the duration of the fire. These calculations show how the fire and smoke propagate through the building and the results were used in the evacuation model for movement of personnel. Personnel start to move towards the exits to find the nearest known exit at the time fire started. All simulation and 3D model data that we used for this study were provided by the Purdue Homeland Security Institute (PHSI).²³ Figure 3 shows the floor plan of The Station nightclub. The 3D model and the background image we used were generated with the same scale and locale as in using the document²¹ from NIST.

Figure 3.

Floor plan of the Station nightclub. Image courtesy: National Institute of Standards and Technology.

Full figure and legend (55K)

Case 2: The second scenario is data recorded at the emergency response training testbed facility that has been developed at Purdue University in a local decommissioned elementary school (Burtsfield Elementary). This scenario is a training exercise in which first responders are called to an in-school shooting incident. Two teams enter the building and attempt to find and apprehend the subject. This data set includes real-time agent location as well as video feeds from both stationary and on-agent cameras. Agents were also equipped with sensor boards providing activity level and directional information. Figure 4 shows the floor plan of the testbed facility.

Figure 4.

Floor plan of the north wing of the Burtsfield Elementary School, West Lafayette, IN.

Full figure and legend (21K)

Visualization for emergency response

Our input data sets have different characteristics that are representative of most emergency-related information. The personnel-related data include moving entity-centered information (e.g., id, location, activity, health level, video), whereas the situational data is global, time-varying information (e.g., fire spread, carbon monoxide distribution). Scene-related data corresponds to a 2D background map and a 3D model.

Personnel-related information

The personnel-related information, including data of moving entities, is displayed using 2D vector graphics. The current position is shown by a circle and the time-evolving path is drawn by line segments. Based on the size of data contained in the circular queue, the paths of the movement is visualized using fading as a method of temporal visualization. The color of each entity's path can be pre-assigned based on entity/team designation, it can be changed based on the entity's health/activity level, or it can be randomly assigned.

To help responders better understand the personnel locations, we have created visual representations of the congestion information and area traveled. Congestion represents how many personnel stay at any position at the same time making it time-based information whereas area traveled is location-based information. Figure 5 shows the visualization of congestion from the fire simulation of case study 1. Congestion is calculated by the number of personnel per unit area (m²) per second. To compute the area traveled, we use the number of personnel that are accumulated at every frame. We divide our environment into a set of cells with each cell representing approximately 1 m² for this scenario. Each moving entity can partially affect congestion based on how much of the entity covers a given cell (e.g., half, quarter). Figure 6 provides a simple example of our congestion calculation for a set of cells and agents. This calculation is based on three propagation areas (e.g., zero propagation area, half propagation area, quarter propagation area), and this example demonstrates how the degree of congestion can propagate to its neighbors. While this is not a completely accurate model, it is reasonable considering the small resolutions and capabilities of mobile devices.

Figure 5.

Congestion visualization. (A) Congestion visualization with personnel. (B) Congestion visualization without personnel.

Full figure and legend (192K)

Figure 6.

Approximate propagation for congestion (9 cells); (red) zero propagation area, (blue) half propagation area, (green) quarter propagation area.

Full figure and legend (64K)

Further personnel-related visualizations include a health or activity-level indicator. In first case study, each entity has two health indicators based on the fractional effective doses (FED) for heat exposure and for gas concentration. Personnel become unconscious and cannot move anymore when either of these values reaches a pre-defined threshold (e.g., threshold=0.5). We nominally visualize healthy personnel as green and display unconscious personnel using red. A health level between 0.0 and 0.5 is visualized with yellow to orangish colors pre-defined in a health level table. Figure 7 (left) shows the evacuation of personnel visualized in a 2D environment for global evacuation analysis. During the visualization, the number of personnel with a given health status is displayed in the information window. After analyzing, it is apparent that 102 of 419 personnel have become incapacitated by the end of the simulation. Figure 7 (right) shows the visualization of moving (circle) and stationary (triangle) entities for second case study.

Figure 7.

Visualization of personnel in 2D environment; (left) 419 people in alive status (green) and their movement path by fading line, (right) 6 moving entities (circle) and 5 stationary entities (triangle).

Full figure and legend (86K)

In the second case study, each entity has a single activity-level indicator based on their amount of movement. As personnel change from a walk to a run, we nominally visualize the slow movements as green and the fast movements as red.

Situation-related information

Situation-related information includes data about the surrounding environment. Our first case study allows to visualize fire simulation data including the temperature, heat release rate (HRR), smoke, CO₂, and CO during the fire as emergency situation-related information. To visualize this information, we use separate 11-element color and gray-level tables. Temperature and HRR are displayed using colors, while smokes, CO₂, and CO are drawn using gray levels. The visualization of each is overlaid on the 2D or 3D environment. Figures 9 and 10 show the results of the visualization of temperature and CO data at different time steps. In Figure 9 and Figure 10, we use a grid of 77 pixel squares to interpolate and visualize temperature and CO since the data was transformed on a coarse grid for performance.

Figure 9.

Temperature distribution at different time steps.

Full figure and legend (124K)

Figure 10.

CO distribution at different time steps.

Full figure and legend (78K)

In the second case study, situational information includes the mobile and stationary camera data streams. Users can select an agent or stationary camera from the application and play the corresponding, synchronized video stream. Figure 8 shows the playback of video data corresponding to a selected agent.

Figure 8.

Video data playback from stationary cameras and agents.

Full figure and legend (211K)

Static scene-related information

Scene-related information includes the building outlines and models. This information plays a serious role in understanding environment, analyzing, and planning for emergency situation. All our visualizations are based on 2D visualization using 2D background map as orthogonal view. In the 2D view, responders can see exit areas and annotation for each exit.

In addition, we provide a 3D perspective view to better understand emergency environment and factors that may have determined the evacuation paths chosen. In the 3D view, all personnel and their environment are visualized as 3D objects. Similarly, 3D navigation and observation can help train first responders by enhancing their recognition of potential evacuation routes and visual building characteristics that may lead responders to probable alternative paths taken by people missing during an actual emergency incident. Particularly, the transparent and wireframe views help responders see personnel behind obstacles. Figures 11 and 12 show the movement of 3D personnel in different shading modes. The transform applied in the 3D view is likewise applied to the 2D view.

Figure 11.

Visualization of personnel and 3D environment.

Full figure and legend (113K)

Figure 12.

Transparent 3D view.

Full figure and legend (102K)

Mobile visual analytics

There is general analytics information that is commonly required for most emergency response situations. Such information helps first responders suggest response priorities and plan actions based on their evaluation of effective situational awareness common operating picture data. Table 1 classifies and lists analytic questions for our emergency cases in terms of personnel and the environment. The most basic information is the location and movement of people and assets (see Figures 7 and 11). In our first scenario, the number of personnel in each health condition (alive, unconscious, dead) and the number of personnel who used each exit are displayed on an information window to analyze the effectiveness of the evacuation. Figure 13 shows a few visual analytics results with such numerical values. When the fire started, most personnel started to run for the exit that was most familiar to them or was closest to them based on the algorithm for personnel's movement used for our first scenario. However, the main exit that might be the most familiar to most people is not most heavily used exit because of congestion nearby the main exit: most personnel ran towards the bar exit. In Figures 9, 10, and 13, we also obtain an unexpected result from the analysis. Although the kitchen area was safer than others were in terms of temperature and carbon monoxide, only a small number of personnel used this area for the evacuation.

Figure 13.

Visualization of personnel-related information: (left) the number of personnel in each health condition, (right) the number of personnel at each exit.

Full figure and legend (93K)

Table 1 - Questions for emergency analysis.

Full table

Figure 14 shows the rate (the number of personnel per second) of evacuation during the fire. The slope of the data line decreased when the main exit became blocked by the crowd. Thus, many personnel chose the other exit (bar exit) for evacuation instead of the main exit. This happened 90 s after the fire occurred. The kitchen exit was not used as an efficient evacuation exit because of its unfamiliarity. Figure 14 also shows the congestion near the main exit that caused the heavy usage of the bar exit.

Figure 14.

Visualization of the rates for evacuation and crowd (total personnel=30).

Full figure and legend (87K)

Figure 15 shows the information of a specific entity selected by a user. The selected entity is displayed in magenta. In Figure 15, the entity with ID=127 and ID=103 became unconscious before evacuation (left). Their FED of CO of the entity is over 0.5, whereas the entity with ID=206 who is still healthy shows the FED of heat and gases at low levels. Moreover, obtaining the change in health levels of each entity at each time can help first responders (e.g., fire fighters) to establish rescue priorities.

Figure 15.

Information of selected entities: (left) for first case study and (right) for second case study.

Full figure and legend (102K)

In the second case study, we can analyze the paths used by the agents and their corresponding video components, and the activity value of each entity is displayed. In Figure 16, we see two agents at the upper right entrance and the corresponding video stream from a stationary camera and an agent in that location. Later in the exercise, we see the agents identifying an unknown entity in the second classroom (Figure 17).

Figure 16.

Agents entering a building: (bottom left) video from stationary camera and (upper left) video from an agent in the location.

Full figure and legend (92K)

Figure 17.

Agents encountering an unknown entity in a building.

Full figure and legend (61K)

Implementation and results

We have implemented and tested our tool on a Dell Axim X51v PDA that uses the Intel 2700G graphics processor and 16 MB of video RAM, an OQO 02 that uses a 1.5 GHz VIA Esther processor with 1 GB of RAM, and a Sprint PCS Vision^SM smart device PPC-6700, which uses Windows Mobile 5.0 and a 416 MHz Intel processor. However, our tool runs on any PDA using Pocket PC with sufficient processing capabilities. We use the Hybrid Rasteroid3 for OpenGL|ES and the OpenVG library provided by Hybrid Graphics, which is a reference implementation and provides functionality through OpenGL|ES 1.1, OpenVG 1.0, and EGL 1.3 specification as announced by the Khronos¹⁶ group. All images in this paper were captured with the Win32 version of our system. Figure 18 shows our system running on mobile devices.

Figure 18.

Photos of our system running on mobile devices: (A) Running on PDA, (B) Running on (left) smartphone and (right) OQO device.

Full figure and legend (215K)

We set the main screen as a 2D orthogonal projection of a building model for global situational awareness since the visualized entities are all in the same 2D plane. In addition, our system does provide 3D perspective views of all the data within the 3D building model. All of the user interfaces are represented with transparency in order to provide a non-invasive interface. The main menu and information window can be also be hidden to not interfere with situational awareness visualizations. As such, this interface can always guarantee a full main view of the situation to the user. Due to the small screen size of mobile devices, the problem of an efficient user interface is another challenge for visualization on mobile devices. GLUT|ES²⁴ has been developed for WinCE and Win32 systems based on OpenGL|ES as the open source implementation. However, it can be space consuming for the visualization of information. Therefore, we implement the API for a user interface based on OpenVG. Currently, it provides a button, a check box, a radio button, a text box, a time slider, a hiding window, a line graph, and vector fonts.

Buttons for play, pause, stop, speed-up/down, and selection mode are provided. The time slider shows the progress of the overall simulation. Menu windows are opened with their own button. Our tool has two menu windows: one is used for displaying text information and the other is used for visualizing additional information, such as rate of evacuation graphs. There are personnel-related sub-menus where a user can choose visualization options related to personnel, and a situation-related menu where a user can select visualization options related to emergency situations, such as viewing a fire spread (e.g., temperature, HRR, CO₂, and CO). In the option menu, a user can toggle 2D/3D views, annotation, and shading modes.

As a prototype mobile visual analytic tools for emergency response and training, our tool presents efficient and interactive visual analytic methods and provides visualization of various types of data. For situations requiring rapid decisions, such as emergency response analysis and services, our system can be used as an efficient testbed.

Based on the overall visualization and analysis for our test datasets, we observed that some personnel evacuated using the stage exit at the beginning of the fire. Most personnel ran about in confusion while they moved to the exits located opposite to the source of the fire near the stage. Some personnel who tried to evacuate out of the main exit failed because of congestion near the corridor between the main exit and the main bar. Hence, many personnel moved to the bar exit. As a result, 102 out of 419 personnel became incapacitated and 97 personnel did not find an evacuation exit. Most of the personnel who became unconscious were found near the exits (e.g., main exit) that may have been familiar with them. In the first case study, at the completion of the simulation, 23% exited via the bar exit, 2% via the kitchen exit, 16% via the main exit, 10% via the stage exit, 15% via the window, and 10% of personnel used a sun-room for their evacuation.

For our second scenario, users felt that the ability to see the coordinated movement of all responding personnel was a great asset in increasing situational awareness and provided needed information for effective decision making. The ability to see video streams from the other team also provides the ability to determine the status and circumstances of unusual events encountered when there is a lack of radio contact and can more efficiently provide situational awareness over general radio traffic.

Capabilities and possibilities for mobile visual analytics for emergency response

Visual analytic systems for emergency response can be used not only during actual emergency events, but also during training, and for hotwash²⁵ and AAR of exercises and incidents. The hotwash is a debriefing and critique conducted immediately after the exercise and incident. The AAR is a detailed and extensive assessment and comment on the exercise with written evaluations that takes place several days or weeks after the exercise. The AAR does not judge success or failure but rather focuses on learning what happened, why things happened, and what tasks and goals were or were not accomplished. Mobile visual analytics adds mobility to common visual analytics and can provide enhanced situational awareness on-site during the hotwash. Such rapid and appropriate awareness leads to 'lessons learned,' which is intended to guide future response direction in order to avoid repeating errors made in the past. Hence, the effectiveness of analysis and evaluation to identify strengths and weaknesses of the response to a given situation can be enhanced.

The capabilities of visual analytics needed for the hotwash include providing integrated visual analytics of additional data. Visual analytics of correlated 2D, 3D, video, and audio data is extremely beneficial in creating lessons learned from exercises and enabling new insight from the interactive exploration and analysis of all information captured during the exercise. Replaying via the time slider and location, by exercise plan as chapters, and at increased/decreased speeds are required. Evaluators and analysts need the ability to display various videos, 2D/3D scene reconstruction, and statistical results of the exercise as they are reviewing performance.

To evaluate the effectiveness and capacity of our system to be used in real emergency situation or emergency training, we received informal feedback from emergency responders in Purdue University fire department and PHSI. Through this feedback, we have learned that our system could be useful in real emergency situations if it is equipped with a real-time tracking capability, since accurate situational awareness is a crucial issue in real situations. Responders also felt that our system will be useful for training such as pre-planning scenarios and site inspections. To this end, we have deployed a version of our system at the Burtsfield Elementary school for use during training sessions.

Conclusion and future work

We have shown a flexible prototype of our mobile visual analytics system for emergency response and demonstrated its use for a building fire evacuation and an exercise for a rescue operation. For situations requiring rapid decisions such as placement and location of public safety assets during a critical incident, our system can be used as an efficient prototype and testbed.

In the future, we will extend this work to include more analytic functions to enhance emergency situational awareness and support rapid decision making. For example, a tool for interactively selecting specific personnel groups and comparing information within and between them can improve analysis of response asset allocation and training effectiveness. Moreover, our system can be extended for actual first responder 3D tracking and visualization for training and in-field deployment support. The integration of RSS data and social network data (e.g., family, friend group, local community, police, fire station, hospital, and government department) will provide interesting visual representation and interrogation challenges, and will further increase the usefulness of our system for emergency response.

References

1. Thomas JJ, Cook KA (Ed). Illuminating the Path: The R&D Agenda for Visual Analytics, IEEE Press: New York, 2005.
2. Kim S, Jang Y, Mellema A, Ebert DS, Collins T. Visual analytics on mobile devices for emergency response. In: VAST '07: Proceedings of IEEE Symposium on Visual Analytics Science and Technology (Sacramento, CA, USA), IEEE Computer Society: Los Alamitos, CA, USA, 2007; 35–42.
3. Fan F, Biagioni ES. An approach to data visualization and interpretation for sensor networks. In: HICSS '04: Proceedings of the 37th Annual Hawaii International Conference on System Sciences (HICSS'04) – Track 3, IEEE Computer Society, Washington, DC, USA, 2004; 30063.1.
4. Koo SO, Kwon HD, Yoon CG, Seo WS, Jung SK. Visualization for a multi-sensor data analysis. In: Proceedings of International Conference on Computer Graphics, Imaging and Visualization (Sydney, Australia), IEEE Computer Society: Washington, DC, USA, 2006; 57–63.
5. Gunnarsson A-S, Rauhala M, Henrysson A, Ynnerman A. Visualization of sensor data using mobile phone augmented reality. In: ISMAR '06: IEEE/ACM International Symposium on Mixed and Augmented Reality (Santa Barbara, USA), IEEE Computer Society: Los Alamitos, CA, USA, 2006; 233–234.
6. Pattath A, Bue B, Jang Y, Ebert DS, Zhong X, Ault A, Coyle E. Interactive visualization and analysis of network and sensor data on mobile devices. In: VAST '06: Proceedings of IEEE Symposium on Visual Analytics Science and Technology (Baltimore, Maryland, USA), IEEE Computer Society: Los Alamitos, CA, USA, 2006; 83–90.
7. Hekmatzada D, Meseth J, Klein R. Non-photorealistic rendering of complex 3d models on mobile devices. In: Proceedings of 8th Annual Conference of the International Association for Mathematical Geology (Berlin, Germany), Alfred-Wegener-Stiftung: Berlin, Germany, vol. 2, 2002; 93–98.
8. Dollner J, Walther M. Real-time expressive rendering of city models. In: Proceedings of the 7th International Conference on Information Visualization (London, UK), IEEE Computer Society: Washington, DC, USA, 2003; 245–250.
9. Diepstraten J, Gorke M, Ertl T. Remote line rendering for mobile devices. In: Proceedings of the Computer Graphics International (Crete, Greece), IEEE Computer Society: Washington, DC, USA, 2004; 454–461.
10. Duguet F, Drettakis G. Flexible point-based rendering on mobile devices. IEEE Computer Graphics and Applications (Los Alamitos, CA, USA), IEEE Computer Society: Los Alamitos, CA, USA, 2004; 24: 57–63.
11. Quillet J-C, Thomas G, Marvie J-E. Client–server visualization of city models through non photorealistic rendering, INRIA Technical Report, September 2005.
12. Pouderoux J, Marvie J-E. Adaptive streaming and rendering of large terrains using strip masks. In: GRAPHITE '05: Proceedings of the 3rd International Conference on Computer Graphics and Interactive Techniques in Australasia and South East Asia, ACM Press: New York, NY, USA, 2005; 299–306.
13. Zhou H, Qu H, Wu Y, Chan M-Y. Volume visualization on mobile devices, PG '06: The 14th Pacific Conference on Computer Graphics and Applications (Taipe, Taiwan), National Taiwan University Press: Taipe, Taiwan, 2006; 76–84.
14. Masoodian M, Budd D. Visualization of travel itinerary information on pdas. In: AUIC '04: Proceedings of the Fifth Conference on Australasian User Interface, (Darlinghurst, Australia), Australian Computer Society, Inc.: Australia, 2004; 65–71.
15. Chittaro L. Visualizing information on mobile devices. Computer 2006; 39: 40–45. | Article |
16. Khronos Group. http://www.khronos.org/ (accessed 30 November 2007).
17. Sanfilippo A, May R, Danielson G, Baddeley B, Riensche R, Kirby S, Collins S, Thornton S, Washington K, Schrager M, Randwyk JV, Borchers B, Gatchell D. An adaptive visual analytics platform for mobile devices. In: SC '05: Proceedings of the 2005 ACM/IEEE Conference on Supercomputing (Washington, DC, USA), IEEE Computer Society: Washington, DC, USA, 2005; 74.
18. Infostar. http://www.sc-conference.org/sc2004/infostar.html (accessed 30 November 2007).
19. Herrero Á, Corchado E, Sáiz JM. Movicab-ids: Visual analysis of network traffic data streams for intrusion detection. In: IDEAL (Burgos, Spain), Springer-Verlag: Berlin, Heidelberg, 2006; 1424–1433.
20. W3C: Scalable Vector Graphics (SVG) XML Graphics for the Web. http://www.w3.org/Graphics/SVG/ (accessed 30 November 2007).
21. Grosshandler WL, Bryner NP, Madrzykowski D, Kuntz K. Report of the technical investigation of the station nightclub fire. [NIST NCSTAR 2: Volume 1] http://www.nist.gov/public_affairs/releases/RI_finalreport_june2905.htm/ (accessed 30 November 2007), June 2005.
22. NIST Fire Dynamics Simulator (FDS) and Smokeview. http://fire.nist.gov/fds/ (accessed 30 November 2007).
23. Chaturvedi A, Mellema A, Filatyev S, Gore J. Dddas for fire and agent evacuation modeling of the rhode island nightclub fire. In: ICCS '06: Workshop on Dynamic Data-Driven Application Systems :(University of Reading, UK). Springer-Verlag: Berlin, Heidelberg, 2006; 433–439.
24. GLUT|ES–The OpenGL|ES Utility Toolkit. http://glutes.sourceforge.net/ (accessed 30 November 2007).
25. Johnson L. Katrina: Hotwash vs. whitewash, [TPMCafe], http://www.tpmcafe.com/story/2005/9/4/111442/8437 (accessed 30 November 2007), September 2005.

Acknowledgements

We thank the Purdue Homeland Security Institute (PHSI) for supplying the simulation data and Army CERDEC for releasing the data used in our second scenario. This work has been funded by the US National Science Foundation (NSF) under Grants 328984 and 0121288, and by the US Department of Homeland Security Regional Visualization and Analytics Center (RVAC) Center of Excellence. We also thank Nicholas Klosterman, Deen King-Smith, and Aravind Mikkilineni for their input.