I’ve written at length about the importance of quality evaluation of exercises. Essentially, if we don’t evaluate exercises, and do it well, the benefits of the exercises are quite limited. Generally, we don’t see a benefit to incidents. By their very nature, incidents threaten and impact life, property, and environment – things we don’t view as being beneficial. However, benefits are often a product of opportunity; and we absolutely should take the opportunity to evaluate our responses.
Many incidents do get evaluated, but through research after the fact. We retrace our steps, review incident documents (such as incident action plans), interview personnel, and examine dispatch logs. These efforts usually paint a decent picture of intent and result (things that are often different), but often miss the delta – the difference between the two – as well as other nuances. When we evaluate an exercise, we do so in real time. Th evaluation effort is best done with preparation. Our evaluation plans, methodologies, and personnel are identified in the design phase of the exercise. Just as we develop emergency operations plans and train personnel to respond, we can develop incident evaluation plans and train personnel to evaluate incident responses.
Understandably, a hurdle we might have is the availability of personnel to dedicate solely to evaluation, especially on larger incidents – but don’t be afraid of asking for mutual aid just to support incident evaluation (just be sure to include them in your preparedness efforts). Just as regional exercise teams should be developed to provide cooperative efforts in exercise design, conduct, and evaluation; incident evaluation teams should be developed regionally. To me, it makes sense for many of these personnel to be the same, as they are already familiar with how to evaluate and write up evaluations.
In exercises, we often use Exercise Evaluation Guides (EEGs) to help focus our evaluation efforts. These are developed based upon identified Core Capabilities and objectives, which are determined early in the exercise design process. While we don’t know the specific objectives we might use in an incident, we can identify these in general, based upon past experiences and our preparedness efforts for future incidents. Similarly, our emergency planning efforts should be based around certain Core Capabilities, which can help inform our incident evaluation preparedness efforts. Job aids similar to EEGs, let’s call them incident evaluation guides (IAGs), can be drafted to prepare for incident evaluation, with adjustments made as necessary when an incident occurs.
Evaluating an incident, in practice, is rather similar to how we would evaluate an exercise, which is why the training for these activities is relatively portable. Evaluation efforts should avoid evaluating individuals, instead focusing on the evaluation of functions and processes. Don’t reinvent the wheel – evaluate based upon documented (hopefully!) plans and procedures and use the Homeland Security Exercise and Evaluation Program (HSEEP) standards to guide your process. Incident evaluation must be managed to ensure that evaluation gaps are minimized and that evaluation progresses as it should. Observations should be recorded and, just as we would for an exercise, prepared for and eventually recorded in an after action report (AAR).
I favor honest after action reports. I’ve seen plenty of after action reports pull punches, not wanting the document to reflect poorly on people. Candidly, this is bullshit. I’ve also heard many legal councils advise against the publication of an after action report at all. Similarly, this is bullshit. If our actions and the need to sustain or improve certain actions or preparations is not properly recorded, necessary changes are much less likely to happen. If an AAR isn’t developed, a corrective action plan certainly won’t be – which gives us no trackable means of managing our improvements and disavows our intent to do so.
As a profession, public safety must always strive to improve. We have plenty of opportunity to assess our performance, not just through exercises, which are valuable, but also through the rigors of incident responses. Prepare for incident evaluation and identify triggers in your emergency plans for when evaluation will be employed, how, and who is involved. Begin evaluation as early as possible in an incident – there are plenty of lessons learned in the early, and often most critical moments of our incident response. Finally, be sure to document lessons learned in an AAR, which will contribute to your overall continuous improvement strategy.
How does your agency accomplish incident evaluation? If you don’t, why?
Need help with the evaluation of incidents? We are happy to help!
© 2017 – Timothy Riecker, CEDP
The Contrarian Emergency Manager 
Tim:
Great point – we all put so much effort into finding ‘value’ from exercises that we forget that the evaluation skills are just as applicable to a live incident or event. I have found that the Safety Officer and their staff are usually the best place to assign evaluation type activities. They are expected to observe as much of the operations as possible so it seems like a common sense application of “other duties as assigned”.
Great idea Aaron!