Dynamic Decision Support for Emergency Responders 2009 Technologies for Homeland Security Conference
Dr. Jill L. Drury (The MITRE Corp.), Dr. Gary L. Klein (The MITRE Corp.), Dr. Mark Pfaff (Indiana U. Indianapolis), and Loretta More (Penn State U.)
Core Idea Let emergency responders visualize more futures and save more lives through Robust Decision Making (RDM)
http://www.surfcityhb.org/images/users/ fire/cedar_fire3a.jpg Photos from Huntingdon Beach (CA) Fire-rescue; right: http://www.surfcityhb.org/images/users/fire/fire_rescue.jpg 2
Problem Decision-making processes break down under complex emergency response situations such as major hurricanes and earthquakes
Hurricane Katrina devastation Photo from http://www.ci.huntington-beach.ca.us/images/users/fire/Hurricane%20Katrina%20 Response2.jpg 3
Robust Decision Making (RDM)
Use simulations to project a landscape of futures Vary things we control Vary things we don’t control
Identify the most robust course of action (COA) COAs evaluated in terms of an aggregated cost measure
Optimal “Optimal” solutions can be so sensitive to small changes in the environment that they can often be poor choices
Robust Robust solutions may not be the best choice under all conditions but are good under most 4
RDM bridges the “Situation Space” and “Decision Space” gap Photo: Jill Drury
Situation space (SS) consists of the facts such as raw sensor data, mapbased information, or alerts
Decision space (DS) involves information fusion or analysis
5
Robust DecisionMaking (RDM) Analysis
Simulation model generates plausible futures for each course of action (COA) and calculates range of costs Decisions involve choosing the most robust COA based on the upper/lower bounds of cost metrics A COA with a tight cost range indicates it is relatively stable even when worst case conditions occur 6
Costs computed by underlying model
Decision-aid software runs a model in the background For each action alternative:
The model looks at everything that could happen in the situation… Will the wind come up or die down? Will it start raining soon?
…and computes the cost for each of those possible futures assuming a given COA
A Huntington Beach (CA) Fire responder surveys the Cedar Fire in 2003. See http://www.surfcity-hb.org/images/users/fire/cedar_fire3a.jpg 7
Example cost measure
Penn State’s NeoCITIES emergency response simulation model adds costs that represent: The number of resources used Property damage Injuries, and/or deaths that might result from a decision
Penn State’s Info Sciences and Technology building
Assigns costs based on the current situation plus extra costs that might be incurred for future incidents if too few resources are conserved to handle them 8
Starting simple: Tukey’s box-plot Depicts a range of costs for each alternative:
The highest cost of all possible futures that might occur The cost of 25% of all futures fall between here and the median The median cost (half cost more & half cost less) The cost of 25% of all futures fall between here and the median The lowest cost of a future under this alternative The costs of 50% of all futures fall within the box
The median cost and range of cost values depend on the likelihood of each possible future and how it will interact with the chosen alternative
Cost
Alternative 1 9
Sending one fire truck will preserve more resources for the future and cost less now So it is has a lower median cost than sending two, and that could be even lower if all external conditions are favorable
Cost Likelihood Decision Will be Regretted
Allows comparison of alternatives’ possible outcomes
1
2 Fire Trucks
10
But there is also an argument for two Sending two fire trucks costs more now but deals better with the worst cases (e.g., high winds); it’s maximum cost is much lower Sending two fire trucks has less downside risk Decision-maker’s role is to choose which alternative is “better” for the situation at hand
Cost Likelihood Decision Will be Regretted
Comparison revisited
1
2 11
Experiment purpose and design
Evaluate the… Impact of RDM information on decision making Use of box-plots for conveying that information Principled design of unambiguous and ambiguous decision-making test situations
Mixed design
“DS group”
“SS group”
Between subjects for box-plots AND text vs. text only Within subjects for ambiguous vs. unambiguous test cases 41 participants
Independent variable: presentation type Primary dependent variables: Chosen course of action Time to make decision Confidence level of decision
12
Post-session questionnaires
Primarily designed to control for possible confounders Three parts drawn from standard instruments: Risk taking versus risk aversion (Blaise and Weber, 2006) Visual versus verbal information processing (Childers et al., 1985) Vivid versus non-vivid imaging (Sheehan, 1967)
Plus… Demographic information, including past experience with emergency response Overall reactions to the degree of decision support provided
13
Test case manipulations
Unambiguous cases Ambiguous cases of type 1: robustness conflicts
Best case vs. worst case vs. median case costs
Cost
Ambiguous cases of type 2: cost function conflicts Type 2A: Magnitude vs. current costs Type 2B: Current costs vs. future costs
14
Test environment
15
Hypotheses
H1: The DS group will make decisions that will result in more positive outcomes H2: The DS group will be more confident in their decisions H3: Non-ambiguous decisions will be made faster than ambiguous decisions by both groups H4: In the case of ambiguous decisions, the DS group will take longer to make the decision H5: The DS group will give higher scores for the degree of decision support provided to them H6: Participants in both groups who believe there is a large chance of future events occurring will under-allocate resources
16
Experiment participants
35 participants from a not-for-profit corporation 13 female, 22 male 14 had previous experience in emergency response 5 were 30 years of age or younger, 7 were 31 – 40, 6 were 41 – 50, 11 were 51 – 60, 6 were 61+
6 participants from a university All 6 were male 2 had previous emergency response experience All 6 were 25 years of age or under
17
Experiment conduct
Participants trained via selfpaced PowerPoint to act as police or fire/rescue commanders Participants got ten practice situations to become familiar with the interface For the main experiment session… Participants provided with text information about a situation One group was also given a graph with box plots for each alternative They chose one alternative and indicated confidence Repeated for 40 situations Then participants answered questions about the experiment and themselves
18
Results: H1
H1: The DS group will make decisions that will result in more positive outcomes. SUPPORTED
DS group made the correct resource allocation 68% of the time, compared to 40% in the SS group (Χ2(1) = 122.99, p < .001) Based on the odds ratio, individuals with the decision support were 3.15 times more likely to get the correct answer than those without
19
Results: H2
H2: The DS group will be more confident in their decisions. SUPPORTED
A one-way Kruskal-Wallis test showed that participants with the decision aid reported much higher confidence (M = 5.41) than those without (M = 4.95), H(1) = 24.11, p < .001 A subsequent Kruskal-Wallis test showed that DS group members also reported much higher confidence for nonambiguous events (M = 5.55) over the other three types (M = 5.07, 5.11, and 4.99), H(3) = 34.12, p < .001
20
Results: H3
H3: Non-ambiguous decisions will be made faster than ambiguous decisions by both groups. SUPPORTED A mixed factorial ANOVA for decision time (R2 = .48) showed that decisions about nonambiguous events were made fastest (M = 41.21 sec) Means for all four types differ significantly at α = .05
60
55 51
50
46 41
40 30 20 10 0
21
Results: H4 and H5
H4: In the case of ambiguous decisions, the DS group will take longer to make the decision. NOT SUPPORTED
A mixed ANOVA showed no significant difference in decision time with respect to condition for any event type (p = .30)
H5: The DS group will give higher scores for the degree of decision support provided to them. SUPPORTED
The DS group rated the system as more highly supportive (M = 5.3) than the SS group (M = 4.5), t(38) = 2.14, p < .05
22
Results: H6
H6: Participants in both groups who believe there is a large chance of future events occurring will under-allocate resources. NOT SUPPORTED Analyzed using one-way withinsubjects ANOVA (R2 = .09) Those reporting a “more than usual” chance of future events over-allocated by .10 resources Those answering “same as usual” or “less than usual” under-allocated by .11 and .09 resources, respectively
0.2 0.1 0 -0.1 -0.2
23
Results: Covariates
No significant effects were found for covariance of any of the following: Prior experience with box-plots Emergency response experience Vivid vs. non-vivid imaging Risk taking versus risk aversion Visual versus verbal information processing A Huntington Beach (CA) fire department member battles the Yorba Linda fire in 2008 (http://www.surfcityhb.org/images/users/fire/berkeley_camera_071.jpg) 24
Conclusions
The decision space information did positively impact decisions made using the box-plot decision aid DS group had a higher confidence in decisions than the SS group DS group felt they had greater decision support than SS group
Participants interact with the NeoCITIES testbed at Penn State (photo courtesy of PSU)
Participants did not appear to have difficulty in understanding or making use of the plots
We successfully introduced decisionmaking conflict based on decision space trade-offs
25
Future work Participants interact with the NeoCITIES testbed at Penn State (photo courtesy PSU)
Further human-subjects experiments: Have participants interact with model’s equation parameters Allow participants to interact with visualizations of results Explore different visualization approaches
26
