Tag AAR

So Your AAR Says Bad Things… Now What?

Tim Riecker, The Contrarian Emergency Manager Leave a comment

There it is.  Your recently delivered after action report (AAR).  Uncomfortably sitting across the room from you.  You eye it like Tom Hanks looking at Wilson for the first time.

Wilson

Wilson!!!

You know what’s in it.  It says bad things.  Things you don’t like.  Things your boss really doesn’t like.  But what will you do?

First, let’s assume that, despite you being unhappy with the areas for improvement identified in the AAR, they are fair representations.  What will you do with the dreaded information now that you have it?  Your AAR may have come with a corrective action plan (CAP), but this is only guidance that still needs to be reviewed and acted upon.

First, each identified area for improvement should be prioritized.  After all, if everything is important, then nothing is important.  Even if the areas for improvement and/or corrective actions are already identified in the AAR (particularly if done by a third party or if the AAR is representative of a multi-agency exercise) you should review this prioritization with your own organization’s stakeholders.  This means pulling together a committee (sorry for cursing!) comprised of key areas within your organization.  This may even mean people from areas that may not have participated, such as information technology, as I’m betting there was something in the exercise about computer systems, programs, internet connection, data access, data continuity, etc.  Don’t forget the finance people, either… some fixes aren’t cheap!

Once everyone has had an opportunity to review the AAR, each identified area for improvement should prioritized, at least to the degrees of high, medium, and low; with a secondary filtering of short-term vs long-term projects.  While some may be relatively quick fixes, others can take months, if not years, to accomplish.  Activities should also be identified that are dependent upon others which may need to be completed first (i.e. a procedure needs to be written before it can be trained on).

That’s probably enough for one meeting.  But the people you gathered aren’t cut loose yet… in fact they are pretty much locked in, so you need to be sure that the people you bring together for this corrective action group have the knowledge, ability, and authority to commit resources within their respective areas of responsibility.  Now that activities have been prioritized, it’s time to assign them… this is why involvement of your boss (if you aren’t the boss) is so important.

Some individuals within your organization will be able to act on their own to make the corrective actions that are needed – while others will need to work together to make these happen.  Consider that there may be more activities than just those identified in the AAR.  For example, the AAR may identify a need for a resource management plan.  That’s good, but we all know you can’t just build a plan and expect it to be ready for action.

For those who are regular readers of my blog, you know I’m a big fan of the POETE elements.  (More on POETE here).  What is POETE?  POETE is an acronym that stands for:

  • Planning
  • Organizing
  • Equipping
  • Training
  • Exercising

What is the value of POETE and what does it all mean?  POETE is a great reminder of the key activities we need to do to enhance our preparedness.  Given that, when we look at an identified need for improvement, we need to consider how to properly address it.  So start at the top:

  • What plans, policies, and procedures are needed to implement and support this corrective action?
  • What organizational impact will occur? Do we need to change our organization in any way?  Do we need to form any special teams or committees to best implement this corrective action?
  • What equipment or systems are needed to support the corrective action?
  • What do people need to be trained in to support the corrective action? Do we need to train them in the plan, about a new policy or procedure?  Do they need training on organizational changes?  How about training in the use of equipment or systems?
  • Lastly, once you’ve made a corrective action, it’s a good idea to test it. Exercises are the best way to accomplish this.

There are obviously other considerations depending on the specific corrective actions and the circumstances of your organization.  Funding is often times one of the most significant.  If you need to obtain funding to make corrective actions, the AAR is one of the best documented investment justifications you can get.

From a project management perspective, the committee should regularly reconvene as a matter of checking in to see how the corrective actions are going.  On a continuing basis, the progress of corrections should be tracked (spreadsheets are great for this), along with who has been tasked with addressing it, timelines for completion, related finances, progress notes, etc.  Otherwise, in our otherwise busy days, these things get lost in the shuffle.

From a program management perspective, this is a process that should be engrained culturally into your organization.  Ideally, one person should be responsible in your organization for coordinating and tracking this corrective action process.  As additional exercises are conducted and actual incidents and events occur, corrective actions from these will be brought into the mix.  It is all too often that organizations complain of seeing the same remarks on every AAR or from experiencing the same issues for every response.  BREAK THE CYCLE!  Establishing a corrective action program for your organization will go a long way toward making these chronic issues go away.

By the way, the same concept can be applied to multi-organizational/agency efforts at any level – local, county, state, federal, regional, etc.  Since we respond jointly, there are great benefits to joint preparedness efforts.  We will likely find that even that we have our own house in order, working with someone else is a very different experience and will require a whole new list of corrective actions as we identify areas for improvement.  This process works great with multi-agency committees.

The bottom line – the biggest reason why we exercise is to test our capabilities.  When we test them, we find faults.  Those faults need to be corrected.  Capitalize on the investment you made in your exercise effort to address those identified deficiencies and improve your capabilities.

What ideas do you have for addressing corrective actions?

Need help with preparedness activities?  Be Proactive and Be Prepared™ – Reach out to Emergency Preparedness Solutions!  We’re always happy to help.

Thanks for reading!

© 2015 – Timothy Riecker

Emergency Preparedness Solutions, LLC

What’s the (next) Big Idea in Emergency Management?

Tim Riecker, The Contrarian Emergency Manager Leave a comment

Innovation.  It seems to be what everyone clamors for.  In emergency management we see people striving for it across the board: in government and in education we try to build the better emergency management mouse trap.  We establish think tanks to find new solutions and the private sector looks for better ways to protect their investments.  But what is it that we are looking for?  What systemic problems do we still face in emergency management that require change? 

There is plenty out there that needs to be improved upon.  There always will be.  Until we can prepare for, prevent, and mitigate disasters to the point that little to no response is ever needed and no loss of life occurs we will continue to strive for better ways of doing things.  I’m guessing that day is a long way off, so we have plenty of work to do.  Before we can innovate, however, we must find cause.  Necessity, as they say, is the mother of invention.  So what needs exist that must be corrected? 

Certainly our after action reports (AARs) identify areas of needed change.  But those generally only show us gaps in local systems.  Threat and Hazard Identification and Risk Analysis (THIRA) likewise shows gaps in local systems.  Does this information ever get fed to higher levels?  Of course it does… in some measure but only some of the time.  States assemble State Preparedness Reports (SPRs) which, in current practice, conduct an analysis of each core capability through each of the POETE elements (planning, organizing, equipping, training, and exercising).  These in turn inform the National Preparedness Report (NPR).  The 2014 NPR was released by FEMA earlier this month, identifying areas for improvement in several of the core capabilities.  This is certainly a resource to help us identify needs, but none of these resources or mechanisms are perfect.  What is missing?  How do we improve them?

Interestingly enough, some opine that we aren’t examining the right data.  The Congressional Research Service suggests that we might need better measures of preparedness, according to their report and this article from FierceHomelandSecurity.com.  The report gives no answers, but poses several questions.  Overall, what can we do better?

Returning to innovation, where do the gaps truly exist?  How do we validate those gaps?  Can we address those gaps with current systems or do we need to create new systems (innovations)?  If it is with current systems, what are the barriers to getting the gaps addressed in the short term?  If it is not with current systems where does the innovation come from? 

Despite having worked in Emergency Management for over fifteen years and having seen, felt, and experienced the myriad changes which have occurred – especially since 9/11 – and with every administration subsequent to the attacks I really hadn’t sat and considered the changes that have occurred.  I’m about half way through an amazing book by John Fass Morton called Next-Generation Homeland Security: Network Federalism and the Course to National Preparedness.  The first 200 pages or so of the book provide a thorough review of civil defense/emergency management/homeland security through decades and over a dozen presidential administrations.  The gravity of it all has left my head spinning.  So many changes – and most simply for the sake of politics.  Much of it seems like wasted effort, but Mr. Morton connects the dots so brilliantly and identifies that D certainly could not have happened if not for A, B, and C… even though C and A were essentially the same.  IT seems that through these years so much has occurred, but so little has actually changed.  I would argue that the practice of emergency management is in a better place now than ever, but what will emergency management look like tomorrow?  Will our continued evolution be through measured change or through innovation?  What makes that determination? 

© 2014 – Timothy Riecker

 

Managing an Exercise Program – Part 10: Evalutation and Improvement Planning

Tim Riecker, The Contrarian Emergency Manager 1 Comment
New HSEEP graphic

New HSEEP graphic

For the previous parts in my series Managing an Exercise Program, please see below:

Managing an Exercise Program – Part 1

Managing an Exercise Program – Part 2: Develop a Preparedness Strategy

Managing an Exercise Program – Part 3: Identify Program Resources and Funding

Managing an Exercise Program – Part 4: Conduct an Annual Training & Exercise Planning Workshop.

Managing an Exercise Program – Part 5: Securing Project Funding

Managing an Exercise Program – Part 6: Conducting Exercise Planning Conferences

Managing an Exercise Program – Part 7: Develop Exercise Documentation

Managing an Exercise Program – Part 8: Preparing Support, Personnel, & Logistical Requirements

Managing an Exercise Program – Part 9: Conducting an Exercise

 

While conducting an exercise is often the most exciting part of the exercise cycle, evaluation and improvement planning lead us to the real reasons why we conduct exercises – to test plans, policies, and procedures.  Evaluation doesn’t just happen.  It must be a deliberate, planned activity within the design and conduct of an exercise.  There are key activities through the design and development of an exercise which link directly to evaluation – starting with the identification of core capabilities to be tested and exercise objectives to be accomplished.  These core capabilities and objectives should lead us directly back to specific plans, policies, and procedures which will be exercised and can help us determine the evaluation methodology and approach.  As we further develop our exercise, be it discussion based or operations based, the decisions we make influence evaluation and the resources needed for it.

When exercise planning begins, we identify staff to fill key roles – including exercise director, simcell manager, lead controller, and lead evaluator.  The lead evaluator doesn’t necessarily have to be involved in all aspects of exercise design, but they need to be informed of key points and should be reviewing draft materials (i.e. the explan or sitman) to become familiar with the details of the exercise and to help them assemble their portion of the control and evaluation plan.  The lead evaluator should ideally be present at exercise planning meetings to become even more familiar with the details, the people, and the facility(s) of exercise play.

Staffing evaluation has a number of options.  Often times evaluators will be selected based on matching their individual areas of expertise with the functional areas of the exercise (i.e. fire, EMS, public health, EOC management, shelter operations, etc.).  You may have a need for more than one evaluator per functional area, particularly if that function is a major component of the exercise and there is a lot to observe.  Remember that if there are multiple exercise venues, they each need to have evaluators assigned who can observe all critical areas.  The bottom line is that evaluation should respect the amount of time, effort, and resources invested in the design and conduct of the exercise.  If the exercise isn’t fairly and accurately evaluated, it’s simply disrespectful to all that effort and those involved.  These concepts apply to both discussion based and operations based exercises.  Members of the exercise planning team should be considered as possible evaluators.

Exercise Evaluation Guides (EEGs) give evaluators guidance in their activity.  The HSEEP website provides a multitude of EEG templates which you should absolutely modify for your own use.  EEGs are provided for each core capability.  Just like planning templates (see my rant about the use of planning templates here), these are guides for you to reach success, but can not and should not be used without modification and customization to ensure that they meet your own specific needs.  EEGs are crafted hand in hand with exercise design to ensure that all bench marks and expected actions are accounted for.  The rating guide on the last page of each EEG is particularly handy to ensure consistency.  You may want to consider including a ‘recommendations’ section for each observation as well, which provides an opportunity for you gain insight from your own subject matter experts on how to address a particular area for improvement.  This will help when it comes to the After Action Report.  I also encourage evaluators to use and ICS form 200 (a blank sheet of paper!) to take some freeform notes.  EEGs, no matter how well designed, simply won’t capture everything.  Personally, when I evaluate, I’ll refer to the EEGs, but make most of my notes on a blank sheet(s) of paper or a steno pad.  I like to mark specific observations, times and timelines, and even inject numbers if I happen to catch them.  After the exercise, I then transcribe my notes into the EEG in the appropriate format.

Training of evaluators is important.  You may have some evaluators which are new and some that are experienced.  Everyone should go through a controller/evaluator briefing prior to the exercise.  Those with little experience might require some extra training, which isn’t a bad thing.  Be sure that both your C/E plan and the briefing identify clearly what you expect from evaluators, how they should conduct themselves, the measure of allowable interaction with players, how they should take notes, and when you expect those notes to be turned in.  The HSEEP website, referenced prior, also has templates for C/E briefings, C/E plans, etc.  You’ve likely heard the saying ‘garbage in – garbage out’… well, exercise evaluation is exactly like that.  If you don’t invest in good training and job aids (EEGs) for your evaluators, you will not get good evaluations!  The Lead Evaluator also needs to lead!  They should check in periodically with each evaluator throughout the exercise to see how things are going and give a quick look at their notes and/or EEGs.  Good feedback will help your evaluators provide better observations.

Finally, at the end of exercise play, I expect evaluators to help with the hotwash.  Sometimes hotwashes are conducted as a single session with all players.  In such sessions, the exercise director or lead evaluator should facilitate and evaluators should take notes.  If the hotwash is tiered, where each functional area will identify their top strengths and areas for improvement; each functional area is facilitated by its evaluator who should also take notes.  This may be followed by a plenary session if possible.  Be sure to have your hotwash strategy planned and identified in the C/E plan!

The After Action Report (AAR) should ideally be written by one or two people based on the EEGs and notes of evaluators.  It may be possible for some observations to be combined, helping to make the document both more concise and easier to read.  Templates are provided on the HSEEP website for AARs and Improvement Plans.  Improvement plans are nothing more than a matrix identifying who will be responsible for what improvement actions, who will assist, what the major benchmarks are to improvement, and when it should be accomplished.  As with the rest of the AAR, these should be drafted and provided prior to the AAR meeting for comment – of which there is usually a great deal.  Once all this is finalized, don’t let this be the last of it!  The implementation of improvements is where organizations often fail!  Rarely do organizations follow through on improvements, resulting in similar observations being noted in exercises and incidents for years after.  The AAR/IP must be championed by someone at an executive level, and coordinated on an ongoing basis by someone who is responsible for tracking progress, coordinating solutions to problems, and reporting progress back to the executive.  For most organizations and jurisdictions, I would suggest a quarterly meeting to review improvement planning progress.  After a few exercises, you may find yourself addressing improvements of several exercises in one meeting.  Accomplishments should be noted, with the opportunity to test the ‘fixes’ in future exercises.  All this feeds back into the strategy planning phase of exercise program management – which is where we started in this series.

This is the end of my ‘Managing an Exercise Program’ series.  I appreciate all those who have read and provided feedback.  Please continue to do so, and best of luck on your next exercise!

I’d be remiss if I didn’t mention that my company – Emergency Preparedness Solutions, LLC – can help you with exercise program planning and management; training and exercise plans; and the design, conduct, and evaluation of exercises; as well as training in HSEEP, exercise design, and exercise evaluation.  Give us a call or email and we’ll be happy to discuss what we can do to help your organization’s preparedness.

EPS logo with tag line - Be proactive, be prepared.

EPS logo with tag line – Be proactive, be prepared.