Tag exercise design

Build a Smart Exercise and Respond to the Unexpected

Tim Riecker, The Contrarian Emergency Manager Leave a comment

A few days ago I caught a documentary on NatGeo called Inside: 21st Century Warship. The documentary was produced in 2013 and told of the cutting-edge design of the USS Freedom and USS Independence. One segment of the documentary captured an exercise the USS Freedom engaged in, with the objective of testing the ship’s firepower to destroy several remote-controlled fast attack boats in open seas. The Captain, well experienced in my opinion, was able to neutralize the boats through the massive wake created by the ship’s sizeable turbines. EndEx.

The lead controller was clearly upset with this. The objective of the exercise, after all, was to test the ship’s guns, which were not fired in this exercise. The controller vented his frustration with the Captain, needing to reemphasize the parameters of the exercise.

Who was at fault in this? Was the objective of the exercise communicated to the Captain? That wasn’t made clear in the documentary. If it was, perhaps it wasn’t made clear that use of the ship’s guns was the only means by which the Captain could engage the attacking boats. I do applaud the Captain’s initial defensive methods, which is perhaps what he was trained to do, though that obviously circumvented the intent of the exercise. Either way, there was a miscommunication or misunderstanding as to the intent and parameters of the exercise.

While this is a military example, the portability to emergency management and homeland security is pretty direct. How do we mitigate against this type of miscommunication or misunderstanding? It starts with a well-defined concept and objectives for our exercise. Those build the foundation from which the rest of the exercise is constructed. Part of exercise design is anticipating how players may respond to the information they are provided and the situations which they will face. This constant analysis helps us to ensure a well-designed exercise, especially in regard to reducing any and all ambiguity, particularly as information relates to the objectives of the exercise and the ‘rules of the game’. It helps us to craft clear injects and even contingency injects in the event players don’t respond the way in which we expect them. Finally, when it comes to deployment of the exercise, an effective player briefing is very important.

Can things still go wrong? Sure they can. That’s why it helps to have a well experienced Exercise Director and/or Lead Controller, and a proficient SimCell Manager (if you are using a SimCell). They can help get the exercise out of a rut. I’ve seen and performed all manner of intervention… most often it’s some ad-hoc development of contingency injects to help steer them down the right path. I’ve also engaged chief executives, who sometimes weren’t expected to participate in the exercise, to make a call, functioning in their own capacity but working for me as an actor, with clear direction to poke, prod, inquire, or otherwise re-direct to get players back into my sandbox. If necessary, it’s a conversation directly with the ‘leader’ of the players, pulling them out of the exercise for a moment and letting them know what they can or can’t be doing. If you have to call a time out and reset something, do it, but do it quickly.

It may be cliché but expect the unexpected. Sometimes players will do something you don’t anticipate. While this may be the circumstance, however, it could very well be on you. Either you didn’t communicate the rules or communicate them well enough. Ensure understanding in this communication. Certainly, ensure that during the exercise, there is good communication between controllers and the SimCell to identify when, if, and how players might be straying a bit. If it’s caught early enough, it will usually just take a gentle nudge to get them back on track. It’s important to recognize and address it as soon as possible – otherwise you will quickly lose your exercise, wasting time and money, and certainly frustrating the players.

Have you had an exercise go off the rails? How did you correct it?

©2020 Timothy Riecker, CEDP

Emergency Preparedness Solutions, LLC®

Improving the HSEEP Templates

Tim Riecker, The Contrarian Emergency Manager Leave a comment

For years it has bothered me that the templates provided for the Homeland Security Exercise and Evaluation Program (HSEEP) are lacking.  The way the documents are formatted and the lack of some important content areas simply don’t do us any favors.  These templates go back to the origination of HSEEP in the early 2000s and they have seen little change since then.  It gives me concern that the people who developed these have struggled with concepts of document structuring and don’t understand the utility of these documents. 

I firmly believe that the documents we use in exercise design, conduct, and evaluation should be standardized.  Many of the benefits of standardization that we (should) practice in the Incident Command System (ICS) certainly apply to the world of exercises, especially when we have a variety of different people involved in each of these key phases of exercises and entering at different times.  Much like an incident, some people develop documents while others are users.  Both should count on a measure of standardization so they don’t have to figure out what they are looking at and how to navigate it before actually diving into the content.  That doesn’t mean, however, that standards can’t evolve to increase utility and function. 

I’ve written in the past about the dangers of templates.  While they are great guides and reminders of certain information that is needed and give us an established, consistent format in which to organize it, I still see too many people not applying some thinking to templates.  They get lost in plugging their information into the highlighted text areas and lose all sense of practicality about why the document is being developed, who the target audience for the document is, and the information they need to convey. 

Some of my bigger gripes…

  • Larger documents, such as ExPlans, SitMans, Controller/Evaluator Handbooks, and After-Action Reports MUST have a table of contents.  These documents can get lengthy and a TOC simply saves time in finding the section you are looking for. 
  • Some exercises are complex and nuanced.  As such, key documents such as ExPlans, SitMans, and Controller/Evaluation Handbooks must have designated space for identifying and explaining those situations.  This could be matters of multiple exercise sites and site-specific information such as different scopes of play for those sites, limited scopes of participation for some agencies, statements on the flow and execution of the exercise, and others.
  • Recognize that the first section of an EEG (Objective, Core Capability, Capability Target, Critical Tasks, and sources) is the only beneficial part of that document.  The next section for ‘observation notes’ is crap.  Evaluators should be writing up observation statements, an analysis of each observation, and recommendations associated with each observation.  The information provided by evaluators should be easily moved into the AAR.  The EEG simply does not facilitate capturing this information or transmitting it to whomever is writing the AAR. 
  • The AAR template, specifically, is riddled with issues. The structure of the document and hierarchy of headings is horrible.  The template only calls for documenting observations associated with observed strengths.  That doesn’t fly with me.  There should similarly be an analysis of each observed strength, as well as recommendations.  Yes, strengths can still be improved upon, or at least sustained.  Big missed opportunity to not include recommendations for strengths.  Further, the narrative space for areas of improvement don’t include space for recommendations.  I think a narrative of corrective actions is incredibly important, especially given the very limited space in the improvement plan; plus the improvement plan is simply intended to be an implementation tool of the AAR, so if recommendations aren’t included in the body of the AAR, a lot is missing for those who want to take a deeper dive and see specifically what recommendations correlate to which observations and with an analysis to support them. 

Fortunately, strict adherence to the HSEEP templates is not required, so some people do make modifications to accommodate greater function.  So long as the intent of each document and general organization remains the same, I applaud the effort.  We can achieve better execution while also staying reasonably close to the standardization of the templates.  But why settle for sub-par templates?  I’m hopeful that FEMA’s National Exercise Division will soon take a look at these valuable documents and obtain insight from benchmark practitioners on how to improve them.  Fundamentally, these are good templates and they have helped further standardization and quality implementation of exercises across the nation.  We should never get so comfortable, though, as to let tools such as these become stagnant, as obsolesce is a regular concern. 

I’m interested in hearing what you have done to increase the value and utility of HSEEP templates.  How would you improve these?  What are your pet peeves? 

© 2020 Timothy Riecker, CEDP

Emergency Preparedness Solutions, LLC®

We’re Off on the Road to Central City

Tim Riecker, The Contrarian Emergency Manager Leave a comment

FEMA Training Bulletin number 1580 announced the publication of the Emergency Management Institute’s (EMI) Online Exercise System Simulation Document (ESSD).  The ESSD, quoted from the website is ‘a compilation of information and resources that would typically be available in any community as they respond to and manage an emergency or disaster in a local community.’  The purpose of the ESSD, essentially, is to provide all the information needed for a fictional jurisdiction for use in exercises.  If you’ve taken some classes at EMI, the HSEEP course, ICS courses, or other programs from FEMA/EMI, you are probably already familiar with the magical place called Central City.

Central City, Liberty County, and other jurisdictions that are part of this fictional area have been in use for decades.  Much of the information published in the ESSD has been available at one point or another, developed to serve the needs of different scenario-based training.  The ESSD packages it all conveniently in one place.

While in some training programs, there is direct benefit to developing a scenario grounded in a real location, we often have course participants who come from different areas.  While you can still certainly develop a scenario in a real location for use in a class like this, the use of Central City (et al) can be an ideal option.  With all the resources and supporting information provided in the ESSD, you likely have everything you need.

Speaking of all that material, what’s in the ESSD?  A sampling:

  • Community profiles
  • Hazard Vulnerability Analysis
  • Laws and Ordinances
  • Emergency Plans
  • Resource and Capability Lists
  • Critical Infrastructure

Having used these jurisdictions and much of this material in courses in the past, course managers do need to expect that it will take time for participants to find some information they are looking for.  In real life, they may or may not be familiar with certain information sets.  The ESSD system was developed to be searchable, which is a huge help, though it’s always good to have one or two paper copies as back ups (note… depending on what you are providing, that may be hundreds of pages).  Since many participants are bringing tablets and laptops to class, all that’s needed is internet service to access all this great information.

I do have a couple of noted observations for improvement.

  1. Having just mentioned printing, I’ll state that first. There doesn’t seem to be a way from within the site to print the material or export it to a PDF.  Yes, you can print from your browser, but formatting is drastically thrown off.
  2. The maps are still horrible. While most of the maps are better than what we’ve had in many of the training materials they have been included in, they are still not high def or zoomable. In fact, many of them are still blurry on my computer screen.  For detailed areas, such as the urban Central City, users (and even scenario designers) may have a need to get much closer to the information.  The Central City map itself, is still difficult to read, especially the myriad of small icons strewn throughout the map.  I would have hoped that re-working this map would have been a priority in the ESSD, as it’s a regular complaint in the classes it’s used in.

Another resource I’ve used in the past (2007 or 2008?) was Zenith City, which was provided by the EPA.  Similar to the FEMA ESSD, the EPA provided a wide array of information for the fictional Zenith City and surrounding jurisdictions.  While the EPA still has a significant exercise program (if you haven’t checked out their stuff, you should), I don’t know whatever happened to the Zenith City information… it’s no where to be found.  (Note: if anyone happens to have the Zenith City material, please let me know!  I’d love to add it back into my library of resources!)

I’m thrilled that FEMA pulled all the Central City material together in this collection, and even added new information.  Central City has always provided us with a great location to unleash hell on, and allow responders and emergency managers to solve so many problems.  I’m just glad I don’t live there!

© 2020 Timothy Riecker, CEDP

Emergency Preparedness Solutions, LLC

Use of Pre-Developed Exercises – Proceed with Caution

Tim Riecker, The Contrarian Emergency Manager 2 Comments

I was recently asked by a client about my thoughts on pre-developed or ‘canned’ exercises.  As it turns out, I have a lot of feelings about them, most of them negative.  Pre-developed exercises, if properly understood and applied, can be a huge help, but the big problem is that we’re dealing with human nature, and some people are just damn lazy.  Garbage in, garbage out.

We need to keep in mind that exercises, fundamentally, are developed to validate plans.  Not my plans.  Your own plans.  While standards of practice mean that most plans have a high degree of commonality (i.e. a HazMat response plan for a jurisdiction in California will be largely the same as one for a jurisdiction in New York State), it’s often the deviations from the standards and the local applications that need to be tested most.  So it doesn’t do well for anyone to replicate an exercise that doesn’t test your own plans.  Similarly, the foundation of exercise design is objectives.  While the pre-developed exercise may have a theme that coincides with what you want to test, sheltering, for example, there are a lot of different aspects of sheltering.  The pre-developed exercise might not focus on what you need to exercise.  With all this, anyone who wants a quality exercise from something pre-developed is going to have to do a lot of re-development, which might be more frustrating than starting from scratch.

HSEEP1

If you want a quality exercise, you really can’t short cut the process.  Not only might HSEEP be required for whatever grant funding you are using for the exercise, but it’s a best practice – and for good reason.  So often people want to cut corners.  If you do, the final product will look like you’ve cut corners.  It might lack proper context, good reference documents, or meaningful evaluation.   The exercise planning meetings have defined purpose, and the documents help capture that process and communicate the intent to specific audiences.

On the other hand, there are proper ways to use materials from a previously developed exercise to benefit your own exercise.  The development of good questions in discussion-based exercises and injects for operations-based exercises can be a challenge.  Reviewing other exercises, especially when there might be some similarity or overlap in objectives, can be a huge help, so long as they are properly contextualized and relate back to objectives for your exercise.  This isn’t a copy and paste, though… as it all should still be applied within the exercise design process.

There are some exercises out there that might seem like exceptions to what I’ve written above.  The first that comes to mind are FEMA’s Virtual Table Top Exercises (VTTX).  The VTTX is a great program, conducted monthly, focusing on different themes and hazards.  FEMA’s Emergency Management Institute (EMI) assembles a package of materials that go to each community registered for the event, allowing a measure of local customization.  While jurisdictions may use this material differently, it is at least an opportunity to discuss relevant topics and hopefully capture some ideas for future implementation.

Similarly, my company, Emergency Preparedness Solutions, recently completed a contract with the Transportation Research Board for a project in which we developed a number of ‘generic’ exercises for airports.  These functional exercises, facilitated through a web-based tool, can be easily customized to meet the needs of most airports across the nation and are written with objectives focused on the fundamentals of EOC management within the timeline of an incident.  While specific plans aren’t directly referenced in the exercises, airport personnel are able to examine the structure of response in their EOC and can reflect on their own plans, policies, and procedures.  Similar to FEMA’s VTTX series, they aren’t a replacement for a custom-developed exercise, but they can help examine some fundamentals and start some important discussions.  I’m not able to get into much more detail on this project, as the final report has yet to be published, but look forward to future posts about it.

All in all, I tend to caution against using pre-developed exercises.  I simply think that most people don’t use them with the right intent and perspective, which can severely limit, or even skew, outcomes.  That said, there exists potential for pre-developed exercises to be properly applied, so proceed with caution and with your wits about you.  Understanding that time, money, and other resources can be scarce, emergency management has always done well with ‘borrowing best practices’.  While there is sometimes nothing wrong with that, short cutting the process will often short cut the benefits.  Do it right.  Use of a custom-developed exercise is going to maximize benefit to your community or organization.

© 2018 – Timothy Riecker, CEDP

EPS New logo

Are You Inviting the Right People to Your Exercises?

Tim Riecker, The Contrarian Emergency Manager Leave a comment

A couple of days ago I started reading Rumsfeld’s Rules – Leadership Lessons in Business, Politics, War, and Life.  Hopefully you have some familiarity with Donald Rumsfeld – the man was a naval aviator, US Congressman, aide to four US presidents, corporate CEO, and is the only person to ever serve as Secretary of Defense twice.  Politics aside, Mr. Rumsfeld has had quite a prolific career.  Throughout this career he has assembled a variety of mantra, proverbs, and sayings which he has used to help guide his career and serve as advice to others.

Early in the book, Mr. Rumsfeld talks about meetings.  What he mentions struck me as solid guidance not only for meetings but also for exercises.  He says “There is a balance that needs to be struck in determining who to invite to a meeting.  You want those who need to be there to contribute substance to the discussion.  But it can be useful to have people who may not be in a position to directly offer substantive input but will benefit from hearing how and why certain decision are being reached.”  Very often exercise offer great opportunity for people to learn – not only the participants but ‘shadowers’ as well.

Mr. Rumsfeld continues on to say “Including a range of people can also ensure that a variety of perspectives will be considered and help identify gaps in information and views.”  Consider that we build, conduct, and evaluate exercises primarily to test plans, polices, and procedures.  This testing is best performed by a spectrum of individuals giving different ideas and perspectives.  Someone may interpret a policy in a completely different way or have an approach to a problem that hasn’t been considered prior.  These fresh ideas, even if flawed, should be brought out into the open for discussion and consideration.

If you’ve followed this blog for any amount of time, you probably know that I prefer smaller meetings and have stressed that participants in exercises should be of a manageable number.  As Mr. Rumsfeld says, there is a balance that must be struck.  You want to be inclusive, but large numbers lend themselves to over-discussion and tangents.  For meetings do you expect the person to add value?  Should they be there given their area of responsibility?  Similarly in exercises is the individual associated with the objectives of the exercise?  (Recall that in exercises we should always reflect on the objectives throughout the entire design process).  When we add more participants to an exercise we need to ensure that they have something to participate in, so injects must be written for them and their activities must be evaluated.

A few years back my team was designing a table top exercise as a lead-in to a significant full-scale exercise.  We did not want to start the full-scale with the initial response, as so many often times are, as the objectives of that exercise were to test the extended response and to examine issues beyond the initial response.  That said, we felt it not fair for us to design such a large exercise by dictating what the first responders would do in the first 48 hours, rather we wanted them to tell us themselves.  So we designed a table top exercise to provide us with their actions both ‘boots on the ground’ as well as policy-level including emergency declarations, evacuation areas, and mutual aid requests.  We were quite fortunate that the design process for the exercise as a whole was very well received and many agencies wanted to participate – from federal, state, county, and local jurisdictions.  The exercise was centered on the state capital, which tends to garner even more attention and participation and included a scenario that most agencies have not participated in prior.  Needless to say, we had a lot of interest.  Nearly every agency invited to the table top wanted to bring not one or two additional people but often times three or four.  We discussed this matter with a few of the key agencies, asking of these were needed participants or observers.  The answer we got was that they were both.  Because of the technical nature of the incident, many agencies realized they needed their main spokesperson supported by one or more technical experts.  We realized this was a fair and reasonable request, but we still needed to figure out how to accommodate them all!

We decided to permit each representative to have a ‘second chair’ – someone seated directly behind them who could advise on technical matters.  Additional specialists were available to them in an adjacent room, which had the discussion live broadcast to them via closed circuit television.  Specialists could be ‘swapped out’ at any time based on the needs of the discussion.  This solution worked well for the exercise, keeping the number of direct participants manageable and meeting the needs of participants to have their specialists available to advise on technical matters – which truly helped inform their decisions and ultimately the outcome of the exercise.

Sometimes, though, you have to say ‘no’.  Realize that as an exercise designer you MUST set a firm deadline on additional participants.  Participants that are added late can set your design team back significantly by needing to ensure that they are written into the exercise and have sufficient activity to make their participation worth while for both them as well as the exercise as a whole – which can be particularly challenging if they are from a different jurisdiction or discipline altogether.  I’ve had to turn down several interested parties and while it’s often difficult to say no, it’s often for the better – and your design team will respect you for it.

What thoughts do you have on ‘right sizing’ your meetings and exercises?  Is there certain guidance that you use?

©2014 Timothy Riecker