By Daryl Lee Spiewak, CEM, TEM, MEP, Lead Trainer for the CEM Commission, and Chair, IAEM-Global Communications Work Group
Last month we discussed the topic of Exercises and Tests with a focus on exercise design and development with an emphasis on improvement planning and the Improvement Plan. This month our emphasis is on how the evaluators determine the areas for improvement that get listed in the Improvement Plan.
Once the exercise is over, the real work of the evaluation team begins. The exercise evaluation team now must analyze all the data collected including that from the HOTWASH, compare performance to the exercise objectives, and evaluate the jurisdiction’s or organization’s ability to meet target capabilities. For USA candidates, this may be recognized as the third step in the Homeland Security Exercise Evaluation Program (HSEEP) exercise evaluation and improvement planning process – Data Analysis.
Data analysis is a four-step process – identifying issues, determining root causes, developing recommendations for improvement, and identifying lessons learned. The results of this four-step process provide the basis for the After-Action Report/Improvement Plan (AAR/IP).
It also recommends, “Evaluators should detail how to implement improvements. They can even recommend who will implement them and provide suggested timeframes for completion.”
All recommendations made should relate to improving performance. Consider policies, plans and procedures; organizational structures; leadership and management processes; training; and resources.
As for the previous standards, NFPA 1600 does not describe how to conduct an emergency management exercise or an Improvement process, so for the exam we need to refer back to various FEMA Independent Study Courses and the HSEEP Manual. Review independent study courses related to exercises such as IS-120a, An Introduction to Exercises, and IS-130, Exercise Evaluation and Improvement Planning, for the core questions. USA candidates also should review the Homeland Security Exercise Evaluation Program (HSEEP) documents.
The application process for both the CEM and the AEM does not require the candidate to address Exercises and Tests specifically as one of the required Knowledge, Skills, and Abilities (KSA) components for the emergency management essay. However, it does require discussion of Prevention, Preparedness, Response, Recovery, and Mitigation activities. In addition, candidates may choose Exercises and Tests as a key component of their Problem Statement and write about that (being sure to cover all the KSAs).
Here are two core-type questions for our analysis in this article.
1. Which of the following describes the goal of exercise data analysis?
a. To allow evaluators to review results of the HOTWASH and participant feedback forms.
b. To demonstrate the successful performance of the exercise objectives.
c. To evaluate the ability of exercised functions to perform target capabilities.
d. To prepare the draft After Action Report and Improvement Plan.
This question is asking you to know why evaluators analyze all the data produced as a result of the exercise. While data analysis does require evaluators to review results of the HOTWASH and participant feedback forms, the Controller/ Evaluator Debriefing is what “allows” them to do so. It is not a goal of exercise data analysis, so response a. is incorrect. In response b., the exercise itself is used to demonstrate performance of exercise objectives, whether successful or not, so this response is incorrect. Response c. is the goal of exercise data analysis. Response d. is also incorrect, because the results of the exercise data analysis are used to prepare the draft documents. Therefore, the correct response is c. See IS 120a, IS-130, and the HSEEP Manual.
2. In what step of the Four-Step Data Analysis process do the evaluators identify discrepancies between what happened and
what was supposed to happen and then explore the source of these discrepancies identified?
a. Identifying issues
b. Determining root causes
c. Developing recommendations for improvement
d. Identifying lessons learned.
This question is asking you to recall what occurs in each of the four data analysis steps. We know that exploring the discrepancies between what happened and what was supposed to happen is a process called “root cause analysis,” the same as response b. Some might consider response a. as correct because evaluators do determine or more correctly document what happened and sometimes what should have happened, but they do not determine the “why it happened” in the first step. Response c. is incorrect because there were no recommended improvements given in the question. The same applies to response d., in that no lessons learned were identified in the question. Therefore, the correct response is b. See IS 120a, IS-130, and the HSEEP Manual.
When reading the questions and responses, be sure you understand exactly what the question is asking of you and read each response before selecting the correct one. It is too easy to get distracted and select a response that appears to be correct but is not the correct response for the question being asked.
Next month we will continue our discussion on Exercises and Tests, with a focus on exercise development and an emphasis on generating Corrective Actions listed in the Improvement Plan. We also will analyze some practice exam questions. As usual, please send any questions you have about the examination or the certification process to me at email@example.com, and I will address them in future articles.
IAEM Bulletin, November 2017
AEM® and CEM® are registered trademarks of the International Association of Emergency Managers.