By Daryl Lee Spiewak, CEM, TEM, MEP, Lead Trainer for the CEM Commission, and Chair, IAEM-Global Communications Work Group

Last month we discussed the topic of Exercises and Tests with a focus on exercise design and development with an emphasis on improvement planning and the Improvement Plan. This month our emphasis is on how the evaluators determine the areas for improvement that get listed in the Improvement Plan.

Exercise Evaluation and the Improvement Planning Process

Once the exercise is over, the real work of the evaluation team begins. The exercise evaluation team now must analyze all the data collected including that from the HOTWASH, compare performance to the exercise objectives, and evaluate the jurisdiction’s or organization’s ability to meet target capabilities. For USA candidates, this may be recognized as the third step in the Homeland Security Exercise Evaluation Program (HSEEP) exercise evaluation and improvement planning process – Data Analysis.

Four-Step Process

Data analysis is a four-step process – identifying issues, determining root causes, developing recommendations for improvement, and identifying lessons learned. The results of this four-step process provide the basis for the After-Action Report/Improvement Plan (AAR/IP).

• Step 1 – Identifying issues. In Step 1 of the process evaluators’ review, the Exercise Evaluation Guide (EEG) analysis sheets are completed for each capability and all its associated tasks. They determine what happened and compare that to what was supposed to happen according to policies, plans, and procedures. They determine the adequacy of resources, mutual aid agreements and contracts. They also determine what the consequences were based on the actions taken and decisions made as well as how well communications, coordination and collaboration worked. Finally, they determine what lessons should be learned and what recommendations could lead to improve- ments or corrective actions to remedy any deficiencies noted.
• Step 2 – Determining root causes. In Step 2, the evaluators identify all the discrepancies between what happened and what was supposed to happen. Then using various tools to analyze these discrepancies, they explore their source or cause. One such method is the “why staircase” described in the FEMA Independent Study course IS-130, Exercise Evaluation and Improvement Planning. Determining the actual root cause of a discrepancy is critical to developing appropriate and effective improvement recommendations. Other methods include cause-and-effect analysis, change analysis, barrier analysis, management oversight analysis, appreciation (So what?) analysis, and risk tree analysis. You may find out how to use these and additional root cause analysis tools in your local library and on the Internet.
• Step 3 – Developing recommendations for improvement. In Step 3, IS-130 states that when developing recommendations for improvement, evaluators should:
  • Identify areas to sustain or improve.
  • Address both short- and long-term solutions.
  • Be consistent with other recommendations.
  • Identify references for implementation.

It also recommends, “Evaluators should detail how to implement improvements. They can even recommend who will implement them and provide suggested timeframes for completion.”

All recommendations made should relate to improving performance. Consider policies, plans and procedures; organizational structures; leadership and management processes; training; and resources.

• Step 4 – Identifying lessons learned. In Step 4, evaluators consider and record lessons learned that could be applied to a similar problem in the future. They also specify what practices should be shared with others in the emergency management community. According to IS-130, “Lessons learned is an innovative practice or a piece of knowledge gained from experience… allowing communities to build on both past experiences and the experiences of one another. For this reason, they save time, conserve money, and accelerate preparedness improvements.” Evaluators ensure that any of these lessons learned that may be applicable to other jurisdictions are “included in the After-Action Report/Improvement Plan (AAR/ IP).”
  
  The Evaluation Team Leader then takes the results of all the data analyses and prepares the draft After Action Report/Improvement Plan (AAR/IP) in preparation for the upcoming an After Action Conference, where the exercise planning team, evaluation team, and other designated stakeholders meet to “review and refine the draft AAR…and to prepare the Improvement Plan.”

Examination References

As for the previous standards, NFPA 1600 does not describe how to conduct an emergency management exercise or an Improvement process, so for the exam we need to refer back to various FEMA Independent Study Courses and the HSEEP Manual. Review independent study courses related to exercises such as IS-120a, An Introduction to Exercises, and IS-130, Exercise Evaluation and Improvement Planning, for the core questions. USA candidates also should review the Homeland Security Exercise Evaluation Program (HSEEP) documents.

Essay

The application process for both the CEM and the AEM does not require the candidate to address Exercises and Tests specifically as one of the required Knowledge, Skills, and Abilities (KSA) components for the emergency management essay. However, it does require discussion of Prevention, Preparedness, Response, Recovery, and Mitigation activities. In addition, candidates may choose Exercises and Tests as a key component of their Problem Statement and write about that (being sure to cover all the KSAs).

Practice Questions

Here are two core-type questions for our analysis in this article.

1. Which of the following describes the goal of exercise data analysis?

a. To allow evaluators to review results of the HOTWASH and participant feedback forms.
b. To demonstrate the successful performance of the exercise objectives.
c. To evaluate the ability of exercised functions to perform target capabilities.
d. To prepare the draft After Action Report and Improvement Plan.

This question is asking you to know why evaluators analyze all the data produced as a result of the exercise. While data analysis does require evaluators to review results of the HOTWASH and participant feedback forms, the Controller/ Evaluator Debriefing is what “allows” them to do so. It is not a goal of exercise data analysis, so response a. is incorrect. In response b., the exercise itself is used to demonstrate performance of exercise objectives, whether successful or not, so this response is incorrect. Response c. is the goal of exercise data analysis. Response d. is also incorrect, because the results of the exercise data analysis are used to prepare the draft documents. Therefore, the correct response is c. See IS 120a, IS-130, and the HSEEP Manual.

2. In what step of the Four-Step Data Analysis process do the evaluators identify discrepancies between what happened and
what was supposed to happen and then explore the source of these discrepancies identified?

a. Identifying issues
b. Determining root causes
c. Developing recommendations for improvement
d. Identifying lessons learned.

This question is asking you to recall what occurs in each of the four data analysis steps. We know that exploring the discrepancies between what happened and what was supposed to happen is a process called “root cause analysis,” the same as response b. Some might consider response a. as correct because evaluators do determine or more correctly document what happened and sometimes what should have happened, but they do not determine the “why it happened” in the first step. Response c. is incorrect because there were no recommended improvements given in the question. The same applies to response d., in that no lessons learned were identified in the question. Therefore, the correct response is b. See IS 120a, IS-130, and the HSEEP Manual.

When reading the questions and responses, be sure you understand exactly what the question is asking of you and read each response before selecting the correct one. It is too easy to get distracted and select a response that appears to be correct but is not the correct response for the question being asked.

Next Month

Next month we will continue our discussion on Exercises and Tests, with a focus on exercise development and an emphasis on generating Corrective Actions listed in the Improvement Plan. We also will analyze some practice exam questions. As usual, please send any questions you have about the examination or the certification process to me at info@iaem.com, and I will address them in future articles.

IAEM Bulletin, November 2017

AEM^® and CEM^® are registered trademarks of the International Association of Emergency Managers.