Managing an Exercise Program – Part 10: Evalutation and Improvement Planning

New HSEEP graphic

New HSEEP graphic

For the previous parts in my series Managing an Exercise Program, please see below:

Managing an Exercise Program – Part 1

Managing an Exercise Program – Part 2: Develop a Preparedness Strategy

Managing an Exercise Program – Part 3: Identify Program Resources and Funding

Managing an Exercise Program – Part 4: Conduct an Annual Training & Exercise Planning Workshop.

Managing an Exercise Program – Part 5: Securing Project Funding

Managing an Exercise Program – Part 6: Conducting Exercise Planning Conferences

Managing an Exercise Program – Part 7: Develop Exercise Documentation

Managing an Exercise Program – Part 8: Preparing Support, Personnel, & Logistical Requirements

Managing an Exercise Program – Part 9: Conducting an Exercise

 

While conducting an exercise is often the most exciting part of the exercise cycle, evaluation and improvement planning lead us to the real reasons why we conduct exercises – to test plans, policies, and procedures.  Evaluation doesn’t just happen.  It must be a deliberate, planned activity within the design and conduct of an exercise.  There are key activities through the design and development of an exercise which link directly to evaluation – starting with the identification of core capabilities to be tested and exercise objectives to be accomplished.  These core capabilities and objectives should lead us directly back to specific plans, policies, and procedures which will be exercised and can help us determine the evaluation methodology and approach.  As we further develop our exercise, be it discussion based or operations based, the decisions we make influence evaluation and the resources needed for it.

When exercise planning begins, we identify staff to fill key roles – including exercise director, simcell manager, lead controller, and lead evaluator.  The lead evaluator doesn’t necessarily have to be involved in all aspects of exercise design, but they need to be informed of key points and should be reviewing draft materials (i.e. the explan or sitman) to become familiar with the details of the exercise and to help them assemble their portion of the control and evaluation plan.  The lead evaluator should ideally be present at exercise planning meetings to become even more familiar with the details, the people, and the facility(s) of exercise play.

Staffing evaluation has a number of options.  Often times evaluators will be selected based on matching their individual areas of expertise with the functional areas of the exercise (i.e. fire, EMS, public health, EOC management, shelter operations, etc.).  You may have a need for more than one evaluator per functional area, particularly if that function is a major component of the exercise and there is a lot to observe.  Remember that if there are multiple exercise venues, they each need to have evaluators assigned who can observe all critical areas.  The bottom line is that evaluation should respect the amount of time, effort, and resources invested in the design and conduct of the exercise.  If the exercise isn’t fairly and accurately evaluated, it’s simply disrespectful to all that effort and those involved.  These concepts apply to both discussion based and operations based exercises.  Members of the exercise planning team should be considered as possible evaluators.

Exercise Evaluation Guides (EEGs) give evaluators guidance in their activity.  The HSEEP website provides a multitude of EEG templates which you should absolutely modify for your own use.  EEGs are provided for each core capability.  Just like planning templates (see my rant about the use of planning templates here), these are guides for you to reach success, but can not and should not be used without modification and customization to ensure that they meet your own specific needs.  EEGs are crafted hand in hand with exercise design to ensure that all bench marks and expected actions are accounted for.  The rating guide on the last page of each EEG is particularly handy to ensure consistency.  You may want to consider including a ‘recommendations’ section for each observation as well, which provides an opportunity for you gain insight from your own subject matter experts on how to address a particular area for improvement.  This will help when it comes to the After Action Report.  I also encourage evaluators to use and ICS form 200 (a blank sheet of paper!) to take some freeform notes.  EEGs, no matter how well designed, simply won’t capture everything.  Personally, when I evaluate, I’ll refer to the EEGs, but make most of my notes on a blank sheet(s) of paper or a steno pad.  I like to mark specific observations, times and timelines, and even inject numbers if I happen to catch them.  After the exercise, I then transcribe my notes into the EEG in the appropriate format.

Training of evaluators is important.  You may have some evaluators which are new and some that are experienced.  Everyone should go through a controller/evaluator briefing prior to the exercise.  Those with little experience might require some extra training, which isn’t a bad thing.  Be sure that both your C/E plan and the briefing identify clearly what you expect from evaluators, how they should conduct themselves, the measure of allowable interaction with players, how they should take notes, and when you expect those notes to be turned in.  The HSEEP website, referenced prior, also has templates for C/E briefings, C/E plans, etc.  You’ve likely heard the saying ‘garbage in – garbage out’… well, exercise evaluation is exactly like that.  If you don’t invest in good training and job aids (EEGs) for your evaluators, you will not get good evaluations!  The Lead Evaluator also needs to lead!  They should check in periodically with each evaluator throughout the exercise to see how things are going and give a quick look at their notes and/or EEGs.  Good feedback will help your evaluators provide better observations.

Finally, at the end of exercise play, I expect evaluators to help with the hotwash.  Sometimes hotwashes are conducted as a single session with all players.  In such sessions, the exercise director or lead evaluator should facilitate and evaluators should take notes.  If the hotwash is tiered, where each functional area will identify their top strengths and areas for improvement; each functional area is facilitated by its evaluator who should also take notes.  This may be followed by a plenary session if possible.  Be sure to have your hotwash strategy planned and identified in the C/E plan!

The After Action Report (AAR) should ideally be written by one or two people based on the EEGs and notes of evaluators.  It may be possible for some observations to be combined, helping to make the document both more concise and easier to read.  Templates are provided on the HSEEP website for AARs and Improvement Plans.  Improvement plans are nothing more than a matrix identifying who will be responsible for what improvement actions, who will assist, what the major benchmarks are to improvement, and when it should be accomplished.  As with the rest of the AAR, these should be drafted and provided prior to the AAR meeting for comment – of which there is usually a great deal.  Once all this is finalized, don’t let this be the last of it!  The implementation of improvements is where organizations often fail!  Rarely do organizations follow through on improvements, resulting in similar observations being noted in exercises and incidents for years after.  The AAR/IP must be championed by someone at an executive level, and coordinated on an ongoing basis by someone who is responsible for tracking progress, coordinating solutions to problems, and reporting progress back to the executive.  For most organizations and jurisdictions, I would suggest a quarterly meeting to review improvement planning progress.  After a few exercises, you may find yourself addressing improvements of several exercises in one meeting.  Accomplishments should be noted, with the opportunity to test the ‘fixes’ in future exercises.  All this feeds back into the strategy planning phase of exercise program management – which is where we started in this series.

This is the end of my ‘Managing an Exercise Program’ series.  I appreciate all those who have read and provided feedback.  Please continue to do so, and best of luck on your next exercise!

I’d be remiss if I didn’t mention that my company – Emergency Preparedness Solutions, LLC – can help you with exercise program planning and management; training and exercise plans; and the design, conduct, and evaluation of exercises; as well as training in HSEEP, exercise design, and exercise evaluation.  Give us a call or email and we’ll be happy to discuss what we can do to help your organization’s preparedness.

EPS logo with tag line - Be proactive, be prepared.

EPS logo with tag line – Be proactive, be prepared.

 

 

Managing an Exercise Program – Part 7: Develop Exercise Documentation

This post is part of a 10-part series on Managing an Exercise Program. In this series I provide some of my own lessons learned in the program and project management aspects of managing, designing, conducting, and evaluating Homeland Security Exercise and Evaluation Program (HSEEP) exercises. Your feedback is appreciated!

Managing an Exercise Program – Part 1

Managing an Exercise Program – Part 2: Develop a Preparedness Strategy

Managing an Exercise Program – Part 3: Identify Program Resources and Funding

Managing an Exercise Program – Part 4: Conduct an Annual Training & Exercise Planning Workshop.

Managing an Exercise Program – Part 5: Securing Project Funding

Managing an Exercise Program – Part 6: Conducting Exercise Planning Conferences

Managing an Exercise Program – Part 7: Develop Exercise Documentation

Managing an Exercise Program – Part 8: Preparing Support, Personnel, & Logistical Requirements

Managing an Exercise Program – Part 9: Conducting an Exercise

Managing an Exercise Program – Part 10: Evaluation and Improvement Planning

 

First I’d like to say that this series of exercise articles has gotten a fair amount of traffic, which I’m quite grateful for.  I’m hopeful that my thoughts and ideas have been able to help those who are looking for experienced insight into emergency management and homeland security exercises.  Certainly if you have anything that you’d like to contribute or have any questions, please post a comment.

We can’t avoid paperwork – ever.  Documentation in exercises, just like in the incident command system (ICS), is a necessity.  Don’t see it as a burden, though, instead view these documents as outcomes of the planning and decision-making process of exercise design.  Just like an incident action plan (IAP) is the result of the planning process in ICS, the primary documents used in exercises (Exercise Plans, Control and Evaluation Plans, Exercise Evaluation Guides, Situation Manual, and Master Scenario Events List) are outcomes of the processes of exercise design.  The graphic below is from HSEEP Volume 2 and provides a quick reference of each document I just listed.  As you will see, each document meets a specific need and is intended for a specific audience.  I will outline some of my tips on each document (except the presentation) below.

Primary HSEEP Documents

Primary HSEEP Documents

Exercise Evaluation Guides (EEGs)  The National Exercise Program provides a variety of EEG templates on their website.  These are an excellent start for your exercise.  Remember that these can and should be customized for your exercise!  While we use capabilities-based exercise planning concepts, and the capabilities are standardized, both the capacity available to anyone in each capability and the means by which a capability is implemented is going to very broadly across the country.  Bottom line: we don’t all do things the same way and we may not be evaluating an entire capability as it’s commonly defined.  EEGs need to be focused on evaluating objectives within given capabilities.  This means that exercise objectives need to be very well-developed to ensure that we are 1) designing an exercise effectively, and 2) evaluating that exercise appropriately.  If we fail in any of these steps (objective development, exercise design, exercise evaluation) we are simply wasting our time.  If need be, draw in your subject matter experts (likely the folks who will be evaluating these areas of the exercise) and get their input on the development of the EEG.  Also consider what the purpose of the EEG is: it helps guide the evaluator in providing constructive commentary on each exercise objective, which will ultimately contribute toward the After Action Report (AAR).

Situation Manual (SitMan)  A SitMan, as stated in the chart above, is used only in discussion-based exercises and is available to all participants.  It should include all information participants need to know to effectively play their role in the exercise.   The most important aspect of this is context and background of the scenario.  Without a well-developed scenario, players have a difficult time getting their ‘head in the game’.  The SitMan will also outline the exercise structure and rules of play, which can vary widely between exercise types (i.e. seminar, workshop, or table top).  Having a good understanding of this information will help players to know what is expected of them.  Be sure to have this (and all) documents reviewed for readability – your focus should be on the audience!  Under most circumstances, the SitMan can be distributed to participants ahead of time.

Controller/Evaluator Plan/Handbook  This document is very audience-focused and as such should very clearly outline the expectations you have of the controllers and evaluators.  It should fully describe their positions, schedules, locations, and scope, as well as expectations.

Exercise Plan (ExPlan)  The ExPlan is often times the core document that everyone wants a copy of – and largely everyone should have access to.  Consider the ExPlan just like the IAP of an incident.  It fully describes what is taking place, where, when, how, and who is involved.  This document will be as complex as your exercise.  For exercises involving multiple venues, each venue should have its own sub-section in the ExPlan describing all the details of what is happening there.

Master Scenario Events List (MSEL)  The MSEL is the script of the exercise.  It should capture everything that is scheduled to occur – from StartEx to EndEx and everything in between.  The bulk of the document is injects, which should be written in detail and carefully reviewed and edited for content and accuracy.  Contingency or back-up injects should also be included but specially indicated as such.  Simulators should keep track of the actual time an inject was performed and what the response was, if any.  This data can be important for both in-exercise follow-up as well as post-exercise evaluation.

Other Documents  Don’t get stuck within the confines of what’s defined by HSEEP.  If you find that you need something else, create it and use it.  I’ve found on several exercises that a very detailed scenario, perhaps even including simulated situation reports and incident action plans is needed.  We’ve come to regard this document as a Ground Truth.  The information in a ground truth doesn’t necessarily need to go to everyone (thus not including it in the ExPlan), especially if the exercise will cover multiple operational periods/shift changes, as the ground truth information is largely only relevant to the starting players.

What tips or experiences do you have with exercise documents?  What other documents have you formulated to meet needs?

Thanks for reading, and be on the look out for Managing an Exercise Program – Part 8: Prepare Support Personnel and Logistical Requirements