Community
By Daryl Lee Spiewak, CEM, TEM, MEP, Lead Trainer for the CEM Commission, and Chair, IAEM-Global Communications Work Group
Last month we discussed the topic of Exercises and Tests with an emphasis on exercise documentation. This month we will continue our focus on exercise design and development with an emphasis on setting up for exercise evaluation.
IS-130a: Exercise Evaluation and Improvement Planning defines exercise evaluation “as the act of reviewing or observing and recording exercise activity or conduct, assessing behaviors or activities against exercise objectives, and noting strengths, weaknesses, deficiencies, or other observations.” It is a process that is more than a single event. Exercise evaluation “should be carefully integrated into the overall exercise design” through the 8-Step Evaluation Process described later in this article.
Our exercise and evaluation program should be designed to evaluate our plans, procedures, training, and organizational capabilities on a schedule determined locally or by other regulatory requirements, with the purpose of establishing and maintaining our required capabilities based on our risk assessments. Both NFPA 1600 (Versions 2013 and 2016) and the FEMA Independent Study Courses IS-120b: An Introduction to Exercises and IS-130a: Exercise Evaluation and Improvement Planning support and even require this of our emergency management programs. We use exercise evaluation to accomplish two specific activities for our emergency management program. First, we assess how well we achieved the exercise objectives. This will be our emphasis today. Second, we identify opportunities to improve our program, which we will emphasize in the next article.
Exercise evaluation assesses our emergency management program at three levels – Mission, Organization/Discipline/Functional, and Task Levels. Which level is chosen for evaluation depends upon the type and complexity of the exercise itself. For example, IS-120a says, “Discussion-based exercises usually focus on Mission Level issues,” where we evaluate plans, policies, procedures and resources. For operations-based exercises, we could focus on one, two, or three level issues and evaluate interagency cooperation, the effectiveness of the organiza-tion’s incident command system, ICS/EOC interface and proper use of equipment.
According to IS-130a, the USA Homeland Security Exercise Evaluation Program standard is “based on capabilities-based planning and emphasizes the need to build capabilities suitable for responding to a wide range of hazards.” It provides that “consistent terminology and tools for all exercise planners can be applied to the full spectrum of hazardous scenarios and incidents” regardless of the country the organization or jurisdiction resides. The eight steps are:
In previous studies, we learned that the Exercise Planning Team develops the capabilities and objectives (using the SMART technique) to be evaluated and then develops the Exercise Evaluation Guides (EEGs). These requirements and supporting documents are based upon the organizational plans, documentation, and personnel needed for the evaluation. They then form the basis for developing the Exercise Evaluation Plan (EvalPlan).
The EvalPlan is a document developed from the exercise requirements established by the Exercise Planning Team. It helps exercise evaluators understand their roles and responsibilities. IS-130a states the EvalPlan also supports the evaluators so they can “conduct an effective analysis of the exercise and produce a comprehensive After-Action Report/Improvement Plan (AAR/IP).”
The EvalPlan isn’t always a separate document. IS-130a tells us, “A Situation Manual (SitMan) and discussion-based evaluation forms” are sometimes substituted in discussion-based exercises. “In operations-based exercises, the EvalPlan is typically part of the Controller/Evaluator Handbook (C/E Handbook).”
The Planning Team also appoints a Lead Evaluator from among the team members. It is best for the Lead Evaluator to be a senior level individual familiar with the issues and objectives associated with the exercise; the organization’s plans, policies, and procedures; the organization’s incident command system and decision-making processes; and the local coordination issues relevant to the exercise. He or she also should possess management analytical skills to lead the evaluation team and analyze the evaluation data to create a final report and improvement plan.
After the planning team determines the exercise evaluation requirements, the Lead Evaluator recruits, assigns, and trains the evaluation team. With the completion of evaluation planning, the Lead Evaluator can finalize the EvalPlan, document it, and distribute it to the evaluators prior to the exercise. Before the exercise actually begins, the Lead Evaluator conducts a briefing for the controllers and evaluators.
As for the previous standards, NFPA 1600 does not describe how to conduct an emergency management exercise, so for the exam we need to refer back to various FEMA Independent Study Courses and the HSEEP Manual. Review independent study courses related to exercises, such as IS-120b: An Introduction to Exercises and IS-130a: Exercise Evaluation and Improvement Planning for the core questions. USA candidates also should review the Homeland Security Exercise Evaluation Program (HSEEP) documents.
The application process for both the CEM® and the AEM® does not require the candidate to address Exercises and Tests specifically as one of the required Knowledge, Skills, and Abilities (KSA) components for the emergency management essay. However, it does require discussion of Prevention, Preparedness, Response, Recovery, and Mitigation activities. In addition, candidates may choose Exercises and Tests as a key component of their Problem Statement and write about that (being sure to cover all the KSAs).
Here are two core-type questions for our analysis in this article.
1. The Exercise Planning Team appoints a Lead Evaluator to organize and manage the exercise evaluation activities. Which of the following people would best fulfill the Lead Evaluator position?
a. Business owner. b. Deputy police chief. c. Mitigation section planner. d. Public works supervisor.
This question is asking you to understand characteristics required of a Lead Evaluator and select the best person who should be qualified to fulfill that position. We know the Lead Evaluator should be a manager skilled with analyzing data. He or she also should be familiar with the organization’s incident command system, policies and procedures, and coordination issues. So by inspection, we can eliminate the business owner and the mitigation section planner, unless the individual possesses the requisite skills. Since we do not know this, they are eliminated from consideration.
The public works supervisor and deputy police chief should have management and analytical skills to perform his or her job. Chances of the public works supervisor possessing knowledge of the organization’s incident command system and coordination issues are extremely low, as they seldom get involved in that level of training or response.
On the other hand, the deputy police chief, by virtue of his or her position, has been involved in all levels of preparedness and various intra- and inter-organizational coordination issues over time. Therefore, given the four limited choices, the correct response is b. See IS 120.b, IS-130a, and the HSEEP Manual.
2. Which statement is true regarding the Exercise Evaluation Plan (EvalPlan)?
a. The EvalPlan is a stand-alone formal document for all types of exercises. b. The EvalPlan is a stand-alone formal document only for discussion-based exercises. c. The EvalPlan is a stand-alone formal document only for operations-based exercises. d. The EvalPlan is a document that helps exercise evaluators understand their roles and responsibilities.
This question is asking you to recall what an Exercise Evaluation Plan (EvalPlan) is and how it is used. We know the EvalPlan could be a separate stand-alone formal-type document, but it doesn’t have to be. It could be incorporated as part of the Situation Manual (SITMAN) in discussion-based exercises or as part of the Controller/Evaluator Handbook (C/E Handbook) in operations-based exercises.Therefore, responses a, b, and c are all untrue. That only leaves response d. as the correct response. See IS 120.b, IS-130a, and the HSEEP Manual.
When reading the questions and responses, be sure you understand exactly what the question is asking of you and read each response before selecting the correct one. It is too easy to get distracted and select a response that appears to be correct but is not the correct response for the question being asked.
Next month we will continue our discussion on Exercises and Tests, with a focus on exercise design and development and an emphasis on conducting the exercise evaluation. We also will analyze some practice exam questions. As usual, please send any questions you have about the examination or the certification process to me at info@iaem.com, and I will address them in future articles.
IAEM Bulletin, September 2017
AEM® and CEM® are registered trademarks of the International Association of Emergency Managers.