It’s Not Too Late To Prepare

The phrase I’ve been using lately when I speak to people has been “It’s not too late to prepare”.  Many people perceive that in the middle of a disaster we are unable to prepare.  Quite the contrary, we have the potential to integrate all of our preparedness steps into a response.  Because we have problems in front of us that need to be addressed, we have an opportunity to continuously improve, ensuring that organizationally we are offering the very best we can. 

There is a reason why there isn’t a mission area for preparedness in the National Preparedness Goal.  This is because preparedness is ongoing.  It’s not a separate or distinct activity.  Rather it is comprised of activities that support all mission areas, no matter when they are actioned.  Preparedness is continuous.

Assessment

Assessment is a key activity within preparedness.  In fact, assessment is foundational in understanding what’s going on.  During a disaster, good management practices dictate that we should be monitoring our response and adjusting as needed.  What exactly should we be monitoring?  Similar to evaluating an exercise, consider the following:

  • What was the effectiveness of deliberate planning efforts? 
    • Were planning assumptions correct?
    • Was the concept of operations adequate in scope and detail? 
    • What was lacking?
    • What worked well?
  • What was the effectiveness of plan implementation?
    • If aspects of plan implementation need improvement, what was the reason for the shortfall?
      • A poor plan
      • Lack of job aids
      • Lack of/poor/infrequent training
      • Lack of practice
      • Lack of the proper resources or capabilities
      • The plan wasn’t followed
  • Did resources and capabilities meet needs?  If not, why?

Planning

While some planning gaps will require a longer time period to address, I’m aware of many jurisdictions and organizations which have been developing plans in the midst of the pandemic.  They recognized a need to have a plan and convened people to develop those plans.  While some of the planning is incident-specific, many of the plans can be utilized in the future we as well, either in the form they were written or adjusted to make them more generally applicable without the specific details of this pandemic.  I’d certainly suggest that any plans developed during the pandemic are reviewed afterwards to identify the same points listed above under ‘assessment’ before they are potentially included in your organization’s catalogue of plans. Also consider that we should be planning for contingencies, as other incidents are practically inevitable.

Training

Training is another fairly easy and often essential preparedness activity which can performed in the midst of a disaster.  Many years ago FEMA embraced the concept of training during disasters.  FEMA Joint Field Offices mobilize with training personnel.  These personnel not only provide just in time training for new personnel or to introduce new systems and processes, but they provide continuing training a variety of topics throughout response and recovery, providing a more knowledgeable workforce.  I’ve seen some EOCs around the country do the same.  Recently, my firm has been contracted to provide remote training for the senior leadership of a jurisdiction on topics such as continuity of operations and multi-agency coordination, which are timely matters for them as they continue to address needs related to the pandemic. 

Exercises

While assessments, planning, and training are certainly activities that may take place during a disaster, exercises are probably less likely, but may, if properly scoped and conducted, still have a place.  Consider that the military will constantly conduct what they call battle drills, even in active theaters of war, to ensure that everyone is familiar with plans and protocols and practiced in their implementation.  Thinking back on new plans that are being written in the midst of the pandemic, it’s a good idea to validate that plan with a tabletop exercise.  We know that even the best written plans will still have gaps that during a blue-sky day we would often identify through an exercise.  Plans written in haste during a crisis are even more prone to have gaps simply because we probably don’t have the opportunity to think everything through and be as methodical and meticulous as we would like.  A tabletop exercise doesn’t have to be complex or long, but it’s good to do a talk through of the plan.  Depending on the scope of the plan and the depth of detail (such as a new procedure, conducting a walk-through of major movements of that plan (that’s a drill) can help ensure validity of the plan and identify any issues in implementation.  While you aren’t likely to go the extent of developing an ExPlan, an evaluator handbook, or exercise evaluation guides (yes, that’s totally OK), it’s still good to lay out a page of essential information to include objectives and methodology since taking the time to write these things down is one more step to ensure that you are doing everything you need for the validation to be effective.  Documentation is still important, and while it can be abbreviated, it shouldn’t be cut out entirely.  It’s also extremely important to isolate the exercise, ensuring that everyone is aware that what is being performed or discussed is not yet part of the response activity.  Evaluators should still give you written observations and documented feedback from participants.  You probably don’t need a full AAR, especially since the observations are going to be put into an immediate modification of the plan in question, but the documentation should still be kept together as there may still be some observations to record for further consideration. 

Evaluation and After Action

Lastly, incident evaluation is something we shouldn’t be missing.  We learn a lot about incident evaluation from exercise evaluation.   I’ve written on it before, which I encourage you to look at, but the fundamentals are ensuring that all actions and decisions are documented, that a hotwash is conducted (or multiple hotwashes to capture larger numbers of people or people who were engaged in very different functions), and that an after action report is developed.   Any incident should provide a lot of lessons learned for your organization, but the circumstances of a pandemic amplify that considerably.  Ensure that everyone in your organization, at all levels, is capturing observations and lessons learned daily.  Ensure that they are providing context to their observations as well, since once this is over, they may not recall the details needed for a recommendation. You may want to consider putting together a short form for people to capture and organize these observations – essentially identifying the issue, providing context, and putting forth a recommendation to address the issue. Don’t forget to encourage people to also identify best practices.  In the end, remember that if lessons learned aren’t actually applied, nothing will change. 

I welcome any insight on how we can continue to apply preparedness in the midst of a disaster. 

Be smart, stay safe, stay healthy, and be good to each other. 

©2020 Timothy Riecker, CEDP

Emergency Preparedness Solutions, LLC

Improving the HSEEP Templates

For years it has bothered me that the templates provided for the Homeland Security Exercise and Evaluation Program (HSEEP) are lacking.  The way the documents are formatted and the lack of some important content areas simply don’t do us any favors.  These templates go back to the origination of HSEEP in the early 2000s and they have seen little change since then.  It gives me concern that the people who developed these have struggled with concepts of document structuring and don’t understand the utility of these documents. 

I firmly believe that the documents we use in exercise design, conduct, and evaluation should be standardized.  Many of the benefits of standardization that we (should) practice in the Incident Command System (ICS) certainly apply to the world of exercises, especially when we have a variety of different people involved in each of these key phases of exercises and entering at different times.  Much like an incident, some people develop documents while others are users.  Both should count on a measure of standardization so they don’t have to figure out what they are looking at and how to navigate it before actually diving into the content.  That doesn’t mean, however, that standards can’t evolve to increase utility and function. 

I’ve written in the past about the dangers of templates.  While they are great guides and reminders of certain information that is needed and give us an established, consistent format in which to organize it, I still see too many people not applying some thinking to templates.  They get lost in plugging their information into the highlighted text areas and lose all sense of practicality about why the document is being developed, who the target audience for the document is, and the information they need to convey. 

Some of my bigger gripes…

  • Larger documents, such as ExPlans, SitMans, Controller/Evaluator Handbooks, and After-Action Reports MUST have a table of contents.  These documents can get lengthy and a TOC simply saves time in finding the section you are looking for. 
  • Some exercises are complex and nuanced.  As such, key documents such as ExPlans, SitMans, and Controller/Evaluation Handbooks must have designated space for identifying and explaining those situations.  This could be matters of multiple exercise sites and site-specific information such as different scopes of play for those sites, limited scopes of participation for some agencies, statements on the flow and execution of the exercise, and others.
  • Recognize that the first section of an EEG (Objective, Core Capability, Capability Target, Critical Tasks, and sources) is the only beneficial part of that document.  The next section for ‘observation notes’ is crap.  Evaluators should be writing up observation statements, an analysis of each observation, and recommendations associated with each observation.  The information provided by evaluators should be easily moved into the AAR.  The EEG simply does not facilitate capturing this information or transmitting it to whomever is writing the AAR. 
  • The AAR template, specifically, is riddled with issues. The structure of the document and hierarchy of headings is horrible.  The template only calls for documenting observations associated with observed strengths.  That doesn’t fly with me.  There should similarly be an analysis of each observed strength, as well as recommendations.  Yes, strengths can still be improved upon, or at least sustained.  Big missed opportunity to not include recommendations for strengths.  Further, the narrative space for areas of improvement don’t include space for recommendations.  I think a narrative of corrective actions is incredibly important, especially given the very limited space in the improvement plan; plus the improvement plan is simply intended to be an implementation tool of the AAR, so if recommendations aren’t included in the body of the AAR, a lot is missing for those who want to take a deeper dive and see specifically what recommendations correlate to which observations and with an analysis to support them. 

Fortunately, strict adherence to the HSEEP templates is not required, so some people do make modifications to accommodate greater function.  So long as the intent of each document and general organization remains the same, I applaud the effort.  We can achieve better execution while also staying reasonably close to the standardization of the templates.  But why settle for sub-par templates?  I’m hopeful that FEMA’s National Exercise Division will soon take a look at these valuable documents and obtain insight from benchmark practitioners on how to improve them.  Fundamentally, these are good templates and they have helped further standardization and quality implementation of exercises across the nation.  We should never get so comfortable, though, as to let tools such as these become stagnant, as obsolesce is a regular concern. 

I’m interested in hearing what you have done to increase the value and utility of HSEEP templates.  How would you improve these?  What are your pet peeves? 

© 2020 Timothy Riecker, CEDP

Emergency Preparedness Solutions, LLC®

Emergency Management and Public Safety Should Prepare Like a Sports Team

When and how did a once-annual exercise become the standard for preparedness?  I suppose that’s fine for a whole plan, but most plans can be carved into logical components that can be not only exercised to various degrees, but training can also be provided to support and compliment each of those components.  There are a lot of elements and activities associated with preparedness.  Consider how sports teams prepare. They are in a constant yet dynamic state of readiness.

Sports teams will review footage of their opponents playing as well as their own games.  We can equate those to reviews of after-action reports, not only of their own performance, but also of others – and with high frequency.  How well does your organization do with this quiz?

  • Do you develop after action reports from incidents, events, and exercises?
  • Are they reviewed with all staff and stakeholders or just key individuals?
  • Are they reviewed more than once or simply archived?
  • Are improvements tracked and reviewed with staff and stakeholders?
  • Do your staff and stakeholders review after action reports from other incidents around the nation?

Planning is obviously important – it’s the cornerstone of preparedness.  Coaches look at standards of practice in the sport, best practices, and maybe come up with their own innovations.  They examine the capabilities of their players and balance those with the capabilities of the opposing team.  They have a standard play book (plan), but that may be modified based upon the specific opponent they are facing.  Their plans are constantly revisited based upon the results of practices, drills, and games.  Plans let everyone know what their role is.

  • Do your plans consider the capabilities of your organization or jurisdiction?
  • Do they truly include the activities needed to address all hazards?
  • Are your plans examined and updated based upon after action reports from incidents, events, and exercises?
  • Are your plans flexible enough for leadership to call an audible and deviate from the plan if needed?
  • Is your organization agile enough to adapt to changes in plans and audibles? How are ad-hoc changes communicated?

Training is a tool for communicating the plan and specific roles, as well as giving people the knowledge and skills needed to execute those roles with precision.  Sports players study their playbooks.  They may spend time in a classroom environment being trained by coaches on the essential components of plays.  Training needs are identified not only from the playbook, but also from after action reviews.

  • Is your training needs-based?
  • How do you train staff and stakeholders to the plan?
  • What training do you provide to help people staffing each key role to improve their performance?

Lastly, exercises are essential.  In sports there are drills and practices.  Drills are used to hone key skill sets (passing, catching, hitting, and shooting) while practices put those skill sets together.  The frequency of drills and practices for sports teams is astounding.  They recognize that guided repetition builds familiarity with plans and hones the skills they learned.  How well do you think a sports team would perform if they only exercised once a year?  So why do you?

  • What are the essential skill sets your staff and stakeholders should be honing?
  • What is your frequency of exercises?
  • Do your exercises build on each other?

I also want to throw in a nod to communication.  Even if you aren’t a sports fan, go attend a local game.  It could be anything… hockey, baseball, soccer, basketball, football… whatever.  It doesn’t necessarily have to be pro.  Varsity, college, or semi-pro would certainly suffice.  Even if you don’t stay for the whole game, there is a lot you can pick up.  Focus on the communication between and amongst players and coaches.  Depending on where you are sitting, you might not be able to hear or understand what they are saying, but what you will notice is constant communication.  Before plays, between plays, and during plays.  Sometimes that communication isn’t just verbal – it might be the tapping of a hockey stick on the ice, clapping of hands, finger pointing, or a hand wave or other silent signal.  Coaches are constantly talking to each other on the bench and with players, giving direction and encouragement.  There is a lot going on… strategy, tactics, offense, defense.  What lessons can you apply to your organization?

Lastly, accomplishments should be celebrated.  In public safety, we tend to ignore a lot of best practices not only of sports teams, but also in general employee relations.  Because of the nature of emergency management and other public safety endeavors, it’s easy to excuse getting stuck in the same rut… we get ready for the next incident, we respond to that incident, and we barely have time to clean up from that incident before the next one comes.  Take a moment to breathe and to celebrate accomplishments.  It’s not only people that need it, but also organizations as a whole.

What lessons can you apply from sports teams to your organization?

© 2019 – Timothy Riecker, CEDP

Emergency Preparedness Solutions, LLC℠

FEMA’s 2017 Hurricane Season AAR

A few days ago, FEMA published its after action report (AAR) for the 2017 hurricane season.  Unless you’ve been living under a rock, you probably know that last year was nothing short of devastating.  The major hurricane activity revolved around Hurricane Harvey (Texas), Hurricane Irma (Caribbean/South Atlantic coast), and Hurricane Maria (Caribbean), but domestic response efforts were also significantly dedicated to a rough season of wildfires in California.  While each of these major disasters was bad enough on its own, the overlap of incident operations between them is what was most crippling to the federal response.  Along with these major incidents were the multitude of typical localized incidents that local, state, and some federal resources manage throughout the year.  2017 was a bad year for disasters.  I don’t think any nation could have supported disaster response as well as the US did.

No response is ever perfect, however, and there were certainly plenty of issues associated with last year’s hurricane responses. Politicians and media outlets made issues in Texas and Puerto Rico very apparent.  While some of these issues may rest on the shoulders of FEMA and other federal agencies, state and local governments hold the major responsibility for them.

This FEMA AAR contains good information, perspective, and reflections.  There are a lot of successes and failures to address.  While I’m not going to write a review of the entire document, which you can read for yourself, but I will discuss a few big-picture items and highlight a few specifics.

First, is the overall organization of the document.  The document is organized through reflection across each of five ‘focus areas’.  I’m not sure why this was the chosen approach.  The doctrinal approach should be a reflection on Core Capabilities, as outlined in the National Preparedness Goal.  Some of these focus areas seem to easily align with a Core Capability, such as ‘Sustained Whole Community Logistics Operations’, which gives me reason to wonder why Core Capabilities were not referenced.  While we use Core Capabilities as a standard in exercises, the purpose for them being part of the National Preparedness Goal is so that we have a standard of reference throughout all preparedness activities.  Any AAR – incident, event, or exercise – should bring us back to preparedness activities.

The second issue I have with the document is the focus.  While it’s understood that this is FEMA’s AAR, not a wholistic federal government AAR, it’s almost too FEMA-centric.  The essence of emergency management is that emergency management agencies are coordination bodies, as such, most of their work gets accomplished through coordinating with other agencies.  While it’s true that FEMA certainly has a significant work force and resources, the AAR seems to stop at the inside threshold of FEMA headquarters, without taking the additional step to acknowledge follow-on actions from a FEMA-rooted issue that may involve other agencies.

Among the positive takeaways were some of the planning assumptions outlined in the report.  There is a short list of planning assumptions on page 9, for example, that provide some encouraging comparisons between planning assumptions and reality.  This is a great reminder for local and state plans to not only include numbers and percentages in their planning assumptions, which will directly lead to identifying capability and resource gaps, but to also reality check those numbers after incidents.

Page 10 of the repost highlights the success of FEMA’s Crisis Action Planning groups.  These groups identified future issues and developed strategies to address these issues.  This is actually an adaptation of an underutilized function within the ICS Planning Section to examine potential medium and long-term issues.

Pages 11 and 12 highlight how Threat and Hazard Identification and Risk Assessment (THIRA) data from states and UASIs can inform response.  It’s encouraging to see preparedness data directly inform response.  I hope this is something that will continue to evolve.

Pages 22 and 23 discuss the staffing issues FEMA had with massive overlapping deployments.  Along with their regular full time workforce, FEMA also deployed a huge volume of their cadre personnel.  They also tapped into a pilot program called State Supplemental Staffing.  While there were some administrative and bureaucratic difficulties, it seems to have been considerably successful.

Overall, this is a good document citing realistic observations and recommendations.  While the document is FEMA-centric, the way of FEMA is the way of emergency management in the US, so it’s always worth keeping an eye on what they are doing, as many of their activities have reach to state and local governments we as other federal agencies.

What important concepts jumped out at you?

© 2018 Timothy Riecker, CEDP

Emergency Preparedness Solutions, LLC™

Hurricane Harvey AAR – Lessons for Us All

Harris County, Texas has recently released their After Action Report (AAR) for Hurricane Harvey that devastated the area last year.  I applaud any AAR released, especially one for an incident of this magnitude.  It requires opening your doors to the world, showing some incredible transparency, and a willingness to discuss your mistakes.  Not only can stakeholders in Harris County learn from this AAR, but I think there are lessons to be learned by everyone in reviewing this document.

First, about making the sausage… The AAR includes an early section on the means and methods used to build the AAR, including some tools provided in the appendix.  Why is this important?  First, it helps build a better context for the AAR and lets you know what was studied, who was included, and how it was pulled together.  Second, it offers a great example for you to use for future incidents.  Developing an AAR for an incident has some significant differences from developing an AAR for an exercise.  Fundamentally, development of an AAR for an exercise begins with design of the exercise and is based upon the objectives identified for that exercise.  For an incident, the areas of evaluation are generally identified after the fact.  These areas of evaluation will focus the evaluation effort and help you cull through the volumes of documentation and stories people will want to tell.  The three focus areas covered in the AAR are Command and Control, Operations, and Mass Care and Sheltering.

Getting into the Harvey AAR itself… My own criticism in the formatting is that while areas for improvement in the AAR follow an Issue/Analysis/Recommendation format, identified strengths only have a sentence or two.  Many AAR writers (for incidents, events, or exercises) think this is adequate, but I do not.  Some measure of written analysis should be provided for each strength, giving it context and describing what worked and why.  I’m also in favor of providing recommendations for identified strengths.  I’m of the opinion that most things, even if done well and within acceptable standards, can be improved upon.  If you adopt this philosophy, however, don’t fall into the trap of simply recommending that practices should continue (i.e. keep doing this).  That’s not a meaningful recommendation.  Instead, consider how the practice can be improved upon or sustained.  Remember, always reflect upon practices of planning, organizing, equipping, training, and exercises (POETE).

As for the identified areas for improvement in AAR, the following needs were outlined:

  • Developing a countywide Continuity of Operations Plan
  • Training non-traditional support personnel who may be involved in disaster response operations
  • Transitioning from response to recovery operations in the Emergency Operations Center
  • Working with the City of Houston to address the current Donations Management strategy

If anything, for these reasons alone, the AAR and the improvement planning matrix attached should be reviewed by every jurisdiction.  Many jurisdictions that I encounter simply don’t have the POETE in place to be successful in addressing these areas.

What is your biggest take away from this AAR?

© 2018 – Timothy Riecker, CEDP

Emergency Preparedness Solutions, LLC™

So Your AAR Says Bad Things… Now What?

There it is.  Your recently delivered after action report (AAR).  Uncomfortably sitting across the room from you.  You eye it like Tom Hanks looking at Wilson for the first time.

Wilson

Wilson!!!

You know what’s in it.  It says bad things.  Things you don’t like.  Things your boss really doesn’t like.  But what will you do?

First, let’s assume that, despite you being unhappy with the areas for improvement identified in the AAR, they are fair representations.  What will you do with the dreaded information now that you have it?  Your AAR may have come with a corrective action plan (CAP), but this is only guidance that still needs to be reviewed and acted upon.

First, each identified area for improvement should be prioritized.  After all, if everything is important, then nothing is important.  Even if the areas for improvement and/or corrective actions are already identified in the AAR (particularly if done by a third party or if the AAR is representative of a multi-agency exercise) you should review this prioritization with your own organization’s stakeholders.  This means pulling together a committee (sorry for cursing!) comprised of key areas within your organization.  This may even mean people from areas that may not have participated, such as information technology, as I’m betting there was something in the exercise about computer systems, programs, internet connection, data access, data continuity, etc.  Don’t forget the finance people, either… some fixes aren’t cheap!

Once everyone has had an opportunity to review the AAR, each identified area for improvement should prioritized, at least to the degrees of high, medium, and low; with a secondary filtering of short-term vs long-term projects.  While some may be relatively quick fixes, others can take months, if not years, to accomplish.  Activities should also be identified that are dependent upon others which may need to be completed first (i.e. a procedure needs to be written before it can be trained on).

That’s probably enough for one meeting.  But the people you gathered aren’t cut loose yet… in fact they are pretty much locked in, so you need to be sure that the people you bring together for this corrective action group have the knowledge, ability, and authority to commit resources within their respective areas of responsibility.  Now that activities have been prioritized, it’s time to assign them… this is why involvement of your boss (if you aren’t the boss) is so important.

Some individuals within your organization will be able to act on their own to make the corrective actions that are needed – while others will need to work together to make these happen.  Consider that there may be more activities than just those identified in the AAR.  For example, the AAR may identify a need for a resource management plan.  That’s good, but we all know you can’t just build a plan and expect it to be ready for action.

For those who are regular readers of my blog, you know I’m a big fan of the POETE elements.  (More on POETE here).  What is POETE?  POETE is an acronym that stands for:

  • Planning
  • Organizing
  • Equipping
  • Training
  • Exercising

What is the value of POETE and what does it all mean?  POETE is a great reminder of the key activities we need to do to enhance our preparedness.  Given that, when we look at an identified need for improvement, we need to consider how to properly address it.  So start at the top:

  • What plans, policies, and procedures are needed to implement and support this corrective action?
  • What organizational impact will occur? Do we need to change our organization in any way?  Do we need to form any special teams or committees to best implement this corrective action?
  • What equipment or systems are needed to support the corrective action?
  • What do people need to be trained in to support the corrective action? Do we need to train them in the plan, about a new policy or procedure?  Do they need training on organizational changes?  How about training in the use of equipment or systems?
  • Lastly, once you’ve made a corrective action, it’s a good idea to test it. Exercises are the best way to accomplish this.

There are obviously other considerations depending on the specific corrective actions and the circumstances of your organization.  Funding is often times one of the most significant.  If you need to obtain funding to make corrective actions, the AAR is one of the best documented investment justifications you can get.

From a project management perspective, the committee should regularly reconvene as a matter of checking in to see how the corrective actions are going.  On a continuing basis, the progress of corrections should be tracked (spreadsheets are great for this), along with who has been tasked with addressing it, timelines for completion, related finances, progress notes, etc.  Otherwise, in our otherwise busy days, these things get lost in the shuffle.

From a program management perspective, this is a process that should be engrained culturally into your organization.  Ideally, one person should be responsible in your organization for coordinating and tracking this corrective action process.  As additional exercises are conducted and actual incidents and events occur, corrective actions from these will be brought into the mix.  It is all too often that organizations complain of seeing the same remarks on every AAR or from experiencing the same issues for every response.  BREAK THE CYCLE!  Establishing a corrective action program for your organization will go a long way toward making these chronic issues go away.

By the way, the same concept can be applied to multi-organizational/agency efforts at any level – local, county, state, federal, regional, etc.  Since we respond jointly, there are great benefits to joint preparedness efforts.  We will likely find that even that we have our own house in order, working with someone else is a very different experience and will require a whole new list of corrective actions as we identify areas for improvement.  This process works great with multi-agency committees.

The bottom line – the biggest reason why we exercise is to test our capabilities.  When we test them, we find faults.  Those faults need to be corrected.  Capitalize on the investment you made in your exercise effort to address those identified deficiencies and improve your capabilities.

What ideas do you have for addressing corrective actions?

Need help with preparedness activities?  Be Proactive and Be Prepared™ – Reach out to Emergency Preparedness Solutions!  We’re always happy to help.

Thanks for reading!

© 2015 – Timothy Riecker

Emergency Preparedness Solutions, LLC

What’s the (next) Big Idea in Emergency Management?

Innovation.  It seems to be what everyone clamors for.  In emergency management we see people striving for it across the board: in government and in education we try to build the better emergency management mouse trap.  We establish think tanks to find new solutions and the private sector looks for better ways to protect their investments.  But what is it that we are looking for?  What systemic problems do we still face in emergency management that require change? 

There is plenty out there that needs to be improved upon.  There always will be.  Until we can prepare for, prevent, and mitigate disasters to the point that little to no response is ever needed and no loss of life occurs we will continue to strive for better ways of doing things.  I’m guessing that day is a long way off, so we have plenty of work to do.  Before we can innovate, however, we must find cause.  Necessity, as they say, is the mother of invention.  So what needs exist that must be corrected? 

Certainly our after action reports (AARs) identify areas of needed change.  But those generally only show us gaps in local systems.  Threat and Hazard Identification and Risk Analysis (THIRA) likewise shows gaps in local systems.  Does this information ever get fed to higher levels?  Of course it does… in some measure but only some of the time.  States assemble State Preparedness Reports (SPRs) which, in current practice, conduct an analysis of each core capability through each of the POETE elements (planning, organizing, equipping, training, and exercising).  These in turn inform the National Preparedness Report (NPR).  The 2014 NPR was released by FEMA earlier this month, identifying areas for improvement in several of the core capabilities.  This is certainly a resource to help us identify needs, but none of these resources or mechanisms are perfect.  What is missing?  How do we improve them?

Interestingly enough, some opine that we aren’t examining the right data.  The Congressional Research Service suggests that we might need better measures of preparedness, according to their report and this article from FierceHomelandSecurity.com.  The report gives no answers, but poses several questions.  Overall, what can we do better?

Returning to innovation, where do the gaps truly exist?  How do we validate those gaps?  Can we address those gaps with current systems or do we need to create new systems (innovations)?  If it is with current systems, what are the barriers to getting the gaps addressed in the short term?  If it is not with current systems where does the innovation come from? 

Despite having worked in Emergency Management for over fifteen years and having seen, felt, and experienced the myriad changes which have occurred – especially since 9/11 – and with every administration subsequent to the attacks I really hadn’t sat and considered the changes that have occurred.  I’m about half way through an amazing book by John Fass Morton called Next-Generation Homeland Security: Network Federalism and the Course to National Preparedness.  The first 200 pages or so of the book provide a thorough review of civil defense/emergency management/homeland security through decades and over a dozen presidential administrations.  The gravity of it all has left my head spinning.  So many changes – and most simply for the sake of politics.  Much of it seems like wasted effort, but Mr. Morton connects the dots so brilliantly and identifies that D certainly could not have happened if not for A, B, and C… even though C and A were essentially the same.  IT seems that through these years so much has occurred, but so little has actually changed.  I would argue that the practice of emergency management is in a better place now than ever, but what will emergency management look like tomorrow?  Will our continued evolution be through measured change or through innovation?  What makes that determination? 

© 2014 – Timothy Riecker

 

Managing an Exercise Program – Part 10: Evalutation and Improvement Planning

New HSEEP graphic

New HSEEP graphic

For the previous parts in my series Managing an Exercise Program, please see below:

Managing an Exercise Program – Part 1

Managing an Exercise Program – Part 2: Develop a Preparedness Strategy

Managing an Exercise Program – Part 3: Identify Program Resources and Funding

Managing an Exercise Program – Part 4: Conduct an Annual Training & Exercise Planning Workshop.

Managing an Exercise Program – Part 5: Securing Project Funding

Managing an Exercise Program – Part 6: Conducting Exercise Planning Conferences

Managing an Exercise Program – Part 7: Develop Exercise Documentation

Managing an Exercise Program – Part 8: Preparing Support, Personnel, & Logistical Requirements

Managing an Exercise Program – Part 9: Conducting an Exercise

 

While conducting an exercise is often the most exciting part of the exercise cycle, evaluation and improvement planning lead us to the real reasons why we conduct exercises – to test plans, policies, and procedures.  Evaluation doesn’t just happen.  It must be a deliberate, planned activity within the design and conduct of an exercise.  There are key activities through the design and development of an exercise which link directly to evaluation – starting with the identification of core capabilities to be tested and exercise objectives to be accomplished.  These core capabilities and objectives should lead us directly back to specific plans, policies, and procedures which will be exercised and can help us determine the evaluation methodology and approach.  As we further develop our exercise, be it discussion based or operations based, the decisions we make influence evaluation and the resources needed for it.

When exercise planning begins, we identify staff to fill key roles – including exercise director, simcell manager, lead controller, and lead evaluator.  The lead evaluator doesn’t necessarily have to be involved in all aspects of exercise design, but they need to be informed of key points and should be reviewing draft materials (i.e. the explan or sitman) to become familiar with the details of the exercise and to help them assemble their portion of the control and evaluation plan.  The lead evaluator should ideally be present at exercise planning meetings to become even more familiar with the details, the people, and the facility(s) of exercise play.

Staffing evaluation has a number of options.  Often times evaluators will be selected based on matching their individual areas of expertise with the functional areas of the exercise (i.e. fire, EMS, public health, EOC management, shelter operations, etc.).  You may have a need for more than one evaluator per functional area, particularly if that function is a major component of the exercise and there is a lot to observe.  Remember that if there are multiple exercise venues, they each need to have evaluators assigned who can observe all critical areas.  The bottom line is that evaluation should respect the amount of time, effort, and resources invested in the design and conduct of the exercise.  If the exercise isn’t fairly and accurately evaluated, it’s simply disrespectful to all that effort and those involved.  These concepts apply to both discussion based and operations based exercises.  Members of the exercise planning team should be considered as possible evaluators.

Exercise Evaluation Guides (EEGs) give evaluators guidance in their activity.  The HSEEP website provides a multitude of EEG templates which you should absolutely modify for your own use.  EEGs are provided for each core capability.  Just like planning templates (see my rant about the use of planning templates here), these are guides for you to reach success, but can not and should not be used without modification and customization to ensure that they meet your own specific needs.  EEGs are crafted hand in hand with exercise design to ensure that all bench marks and expected actions are accounted for.  The rating guide on the last page of each EEG is particularly handy to ensure consistency.  You may want to consider including a ‘recommendations’ section for each observation as well, which provides an opportunity for you gain insight from your own subject matter experts on how to address a particular area for improvement.  This will help when it comes to the After Action Report.  I also encourage evaluators to use and ICS form 200 (a blank sheet of paper!) to take some freeform notes.  EEGs, no matter how well designed, simply won’t capture everything.  Personally, when I evaluate, I’ll refer to the EEGs, but make most of my notes on a blank sheet(s) of paper or a steno pad.  I like to mark specific observations, times and timelines, and even inject numbers if I happen to catch them.  After the exercise, I then transcribe my notes into the EEG in the appropriate format.

Training of evaluators is important.  You may have some evaluators which are new and some that are experienced.  Everyone should go through a controller/evaluator briefing prior to the exercise.  Those with little experience might require some extra training, which isn’t a bad thing.  Be sure that both your C/E plan and the briefing identify clearly what you expect from evaluators, how they should conduct themselves, the measure of allowable interaction with players, how they should take notes, and when you expect those notes to be turned in.  The HSEEP website, referenced prior, also has templates for C/E briefings, C/E plans, etc.  You’ve likely heard the saying ‘garbage in – garbage out’… well, exercise evaluation is exactly like that.  If you don’t invest in good training and job aids (EEGs) for your evaluators, you will not get good evaluations!  The Lead Evaluator also needs to lead!  They should check in periodically with each evaluator throughout the exercise to see how things are going and give a quick look at their notes and/or EEGs.  Good feedback will help your evaluators provide better observations.

Finally, at the end of exercise play, I expect evaluators to help with the hotwash.  Sometimes hotwashes are conducted as a single session with all players.  In such sessions, the exercise director or lead evaluator should facilitate and evaluators should take notes.  If the hotwash is tiered, where each functional area will identify their top strengths and areas for improvement; each functional area is facilitated by its evaluator who should also take notes.  This may be followed by a plenary session if possible.  Be sure to have your hotwash strategy planned and identified in the C/E plan!

The After Action Report (AAR) should ideally be written by one or two people based on the EEGs and notes of evaluators.  It may be possible for some observations to be combined, helping to make the document both more concise and easier to read.  Templates are provided on the HSEEP website for AARs and Improvement Plans.  Improvement plans are nothing more than a matrix identifying who will be responsible for what improvement actions, who will assist, what the major benchmarks are to improvement, and when it should be accomplished.  As with the rest of the AAR, these should be drafted and provided prior to the AAR meeting for comment – of which there is usually a great deal.  Once all this is finalized, don’t let this be the last of it!  The implementation of improvements is where organizations often fail!  Rarely do organizations follow through on improvements, resulting in similar observations being noted in exercises and incidents for years after.  The AAR/IP must be championed by someone at an executive level, and coordinated on an ongoing basis by someone who is responsible for tracking progress, coordinating solutions to problems, and reporting progress back to the executive.  For most organizations and jurisdictions, I would suggest a quarterly meeting to review improvement planning progress.  After a few exercises, you may find yourself addressing improvements of several exercises in one meeting.  Accomplishments should be noted, with the opportunity to test the ‘fixes’ in future exercises.  All this feeds back into the strategy planning phase of exercise program management – which is where we started in this series.

This is the end of my ‘Managing an Exercise Program’ series.  I appreciate all those who have read and provided feedback.  Please continue to do so, and best of luck on your next exercise!

I’d be remiss if I didn’t mention that my company – Emergency Preparedness Solutions, LLC – can help you with exercise program planning and management; training and exercise plans; and the design, conduct, and evaluation of exercises; as well as training in HSEEP, exercise design, and exercise evaluation.  Give us a call or email and we’ll be happy to discuss what we can do to help your organization’s preparedness.

EPS logo with tag line - Be proactive, be prepared.

EPS logo with tag line – Be proactive, be prepared.