Are You Inviting the Right People to Your Exercises?

A couple of days ago I started reading Rumsfeld’s Rules – Leadership Lessons in Business, Politics, War, and Life.  Hopefully you have some familiarity with Donald Rumsfeld – the man was a naval aviator, US Congressman, aide to four US presidents, corporate CEO, and is the only person to ever serve as Secretary of Defense twice.  Politics aside, Mr. Rumsfeld has had quite a prolific career.  Throughout this career he has assembled a variety of mantra, proverbs, and sayings which he has used to help guide his career and serve as advice to others.

Early in the book, Mr. Rumsfeld talks about meetings.  What he mentions struck me as solid guidance not only for meetings but also for exercises.  He says “There is a balance that needs to be struck in determining who to invite to a meeting.  You want those who need to be there to contribute substance to the discussion.  But it can be useful to have people who may not be in a position to directly offer substantive input but will benefit from hearing how and why certain decision are being reached.”  Very often exercise offer great opportunity for people to learn – not only the participants but ‘shadowers’ as well.

Mr. Rumsfeld continues on to say “Including a range of people can also ensure that a variety of perspectives will be considered and help identify gaps in information and views.”  Consider that we build, conduct, and evaluate exercises primarily to test plans, polices, and procedures.  This testing is best performed by a spectrum of individuals giving different ideas and perspectives.  Someone may interpret a policy in a completely different way or have an approach to a problem that hasn’t been considered prior.  These fresh ideas, even if flawed, should be brought out into the open for discussion and consideration.

If you’ve followed this blog for any amount of time, you probably know that I prefer smaller meetings and have stressed that participants in exercises should be of a manageable number.  As Mr. Rumsfeld says, there is a balance that must be struck.  You want to be inclusive, but large numbers lend themselves to over-discussion and tangents.  For meetings do you expect the person to add value?  Should they be there given their area of responsibility?  Similarly in exercises is the individual associated with the objectives of the exercise?  (Recall that in exercises we should always reflect on the objectives throughout the entire design process).  When we add more participants to an exercise we need to ensure that they have something to participate in, so injects must be written for them and their activities must be evaluated.

A few years back my team was designing a table top exercise as a lead-in to a significant full-scale exercise.  We did not want to start the full-scale with the initial response, as so many often times are, as the objectives of that exercise were to test the extended response and to examine issues beyond the initial response.  That said, we felt it not fair for us to design such a large exercise by dictating what the first responders would do in the first 48 hours, rather we wanted them to tell us themselves.  So we designed a table top exercise to provide us with their actions both ‘boots on the ground’ as well as policy-level including emergency declarations, evacuation areas, and mutual aid requests.  We were quite fortunate that the design process for the exercise as a whole was very well received and many agencies wanted to participate – from federal, state, county, and local jurisdictions.  The exercise was centered on the state capital, which tends to garner even more attention and participation and included a scenario that most agencies have not participated in prior.  Needless to say, we had a lot of interest.  Nearly every agency invited to the table top wanted to bring not one or two additional people but often times three or four.  We discussed this matter with a few of the key agencies, asking of these were needed participants or observers.  The answer we got was that they were both.  Because of the technical nature of the incident, many agencies realized they needed their main spokesperson supported by one or more technical experts.  We realized this was a fair and reasonable request, but we still needed to figure out how to accommodate them all!

We decided to permit each representative to have a ‘second chair’ – someone seated directly behind them who could advise on technical matters.  Additional specialists were available to them in an adjacent room, which had the discussion live broadcast to them via closed circuit television.  Specialists could be ‘swapped out’ at any time based on the needs of the discussion.  This solution worked well for the exercise, keeping the number of direct participants manageable and meeting the needs of participants to have their specialists available to advise on technical matters – which truly helped inform their decisions and ultimately the outcome of the exercise.

Sometimes, though, you have to say ‘no’.  Realize that as an exercise designer you MUST set a firm deadline on additional participants.  Participants that are added late can set your design team back significantly by needing to ensure that they are written into the exercise and have sufficient activity to make their participation worth while for both them as well as the exercise as a whole – which can be particularly challenging if they are from a different jurisdiction or discipline altogether.  I’ve had to turn down several interested parties and while it’s often difficult to say no, it’s often for the better – and your design team will respect you for it.

What thoughts do you have on ‘right sizing’ your meetings and exercises?  Is there certain guidance that you use?

©2014 Timothy Riecker

Business Continuity – More than just a plan

Don't throw away all of your effort to build your business - Be prepared!  (image courtesy of FEMA)

Don’t throw away all of your effort to build your business – Be prepared! (image courtesy of FEMA)

Every year businesses are forced to close due to the impacts of disaster.  Research from the National Federation of Independent Business (NFIB) tells us that the top four threats to business suffering the impact of disaster are:

  1. Power Loss
  2. Loss of Sales and Customers
  3. Length of Recovery
  4. Uninsured Loss

How can businesses protect themselves against these impacts?  Planning for them is, of course, the easy answer.  Just like governments, though, wouldn’t it make the most sense for a business to have an emergency preparedness program in place?

Consider that small business owners invest a great deal of time, energy, and funding to build and grow their business.  As an independent consultant I can be working on a variety of things on any given day including project management, marketing, and accounting.  Small business owners that deal with products (vs services) often times have even more to deal with including inventory, vendors, and distributors.  The foundation of these entrepreneurial efforts is often times the business plan.  Aspiring business owners put a lot of effort into creating this plan which describes what the business will do, what the market capacity is, what the competition looks like, and even trying to forecast revenues for several years.  A successful business may continue elements of this business plan years later through a strategic plan intended to guide growth and company-wide efforts.  Doesn’t it make sense that if we put so much effort into building and growing our businesses that we put some effort into ensuring that our businesses will survive a disaster?

As a society we generally like plans.  They are an organized tome capturing our assumptions, ideas, and strategies to accomplish something.  Plans are good and certainly help us through a great deal.  A disaster plan, though, is not a disaster program.  The plan may embody our program, helping to guide and inform our decisions in the event of a disaster, but our preparedness efforts must stretch beyond a plan if we are to be successful.  Consider DHS’ POETE capability elements – Planning, Training, Organizing, Equipping, Training, and Exercising.  With these elements in your head scroll back up to those top four threats from the NFIB and give them a moment of thought.  You probably now have some additional ideas as to how you can address and prevent each of those with activity which may go beyond planning.

This recent article from Small Business Trends (which provided my initial inspiration for this blog post) provides a good outline of initial considerations for every business relative to disaster preparedness.

What does your business do to be better prepared?

Shameless plug time: Need help building your business continuity program?  Emergency Preparedness Solutions can help!  Contact us at consultants@epsllc.biz or check out our website at www.epsllc.biz.

© 2014 Timothy Riecker

 

Hackers Endanger Public Safety With Pranks

VMS Vulnerabilities Can Have Serious Consequences (Image from Slate.com)

VMS Vulnerabilities Can Have Serious Consequences (Image from Slate.com)

Over the past few years we’ve seen some prominent occurrences of hackers gaining access to public safety systems where they make changes which, while a bit humorous at first blush, are serious examples of the vulnerability of our systems.

This article describes a vulnerability in variable message signs (VMS), which can be programmed remotely to notify drivers of hazards or give other pertinent information.  In another occurrence, in February of 2013, hackers gained access to the Emergency Alert System, broadcasting messages about a zombie attack.

The favor these pranks do for us is to identify vulnerabilities in our systems.  Both articles mention that some vulnerabilities were exploited simply because the default passwords on these systems were never changed.  Agencies that maintain any kind of public messaging system (and yes, this should also include websites and social media accounts), should adhere to the guidance we all normally hear about passwords – create strong passwords including combinations of numbers, letters, and symbols (when possible), avoid patterns or predictable passwords, and change passwords regularly.  As a matter of information security, these passwords should only be known by a select few.

Why are these occurrences serious?  Obviously (to most of us) they are taken in jest, but these are public safety systems which should only be accessed by public safety professionals.  The information and instructions provided over these systems need to come from reliable sources to ensure that the public takes the messages seriously and follows the instructions given.  We should be thankful these instances were pranks, as someone with malicious intent could have provided information which could have endangered the public.

All levels of government and any other organizations which maintain public alerting systems, including colleges and universities and even highway construction firms need to make a thorough examination of their systems, identify potential vulnerabilities, and take steps to ensure they are protected.

What other systems offer vulnerabilities to hacking?

 

© 2014 Timothy Riecker

 

 

Engaging a Nation in Preparedness – Learning from History

June 14, 1954 saw the first nation-wide civil defense drill conducted in the United States.  The Civil Defense Administration organized and promoted the event, which included operations in 54 cities around the country, including Puerto Rico, the US Virgin Islands, Alaska, and Hawaii.  Canada had also participated in the event.  The History Channel website has a nice write-up on the event.  The History Channel’s article explains some of the activities conducted during the event, which largely consisted of sheltering drills.

Today we do see some nation-wide exercises which engage citizens through the Shake Out earthquake drills.  Their website has a great deal of information on the program, including how you can participate.  The statistics on their site are great, showing not only the US regional exercises but also Shake Out exercises conducted in nations around the world (something I was not familiar with until visiting their site this morning).  The earthquake hazard in the US and around the globe is significant – in fact we just saw two large earthquakes late yesterday – a 7.9 near Alaska and a 7.2 near New Zealand.  While the core activity of the Shake Out exercises is the ‘Drop, Cover, and Hold on’ (similar to the ‘duck and cover’ of the civil defense days), their website also promotes preparedness activities including a ‘hazard hunt’ for items which may fall during an earthquake, a family disaster plan, business and organizational continuity planning, and emergency supply kits.  This is the type of preparedness activity we need to continue, but we also need to do more.  Unfortunately the message still isn’t getting through to many people.

How do you think we should get the message out?

 

© 2014 Timothy Riecker

 

 

Hazard Analysis – Looking Beyond Your Borders

In the radiological emergency preparedness niche field of emergency management we conduct a lot of preparedness activities for a hazard which may not even be within our jurisdiction.  The emergency planning zone (EPZ) for a nuclear power plant often times transcends multiple towns, cities, villages, counties, and even state lines.  While I have some issues with the effectiveness and implementation of radiological emergency planning, they at least address the reality of the hazard crossing the artificial borders we humans have established.  For other hazards, this premise usually does not hold true.

In January of this year a chemical leaked from a storage tank at a coal processing facility in Charleston, West Virginia.  This chemical leaked into the Elk River and both directly and indirectly impacted hundreds of thousands of citizens, businesses, and governments requiring evacuations and preventing water use for several weeks. The DHS Lessons Learned Information Sharing (LLIS) website has posted a brief by The Joint Commission on this incident with specific citations on the impacts to area hospitals, mostly through contracted laundry services.

In the private sector, we often encourages businesses to examine the vulnerabilities of suppliers and distributors as part of their hazard vulnerability analysis (HVA) and business impact assessment (BIA).  This is not something often considered by governments.  For example, in my town, there is only one very small gas station, so due their limited hours (fuel is not available 24/7) government services and the town’s contracted fire company must leave the town for fuel.  That is a significant dependency on a supplier outside the jurisdiction.  I’ve sure there are many other suppliers used by the town which lie outside their borders.  Additionally, what are the potential impacts of an incident that occurs in a neighboring jurisdiction?  Such an incident could either directly impact you, such as a chemical plume entering your jurisdiction; or would require your jurisdiction to address sheltering, traffic, or mutual aid needs.

I would suggest, as part of the hazard analysis phase of your planning process, that you obtain copies of the hazard analysis of neighboring jurisdictions.  The hazards they indicate may be quite eye-opening to you and may require you to better prepare for a hazard beyond your borders.

©2014 Timothy Riecker

Talking Turkey – Point of Distribution (POD) Exercise

I recently read an article (although I can’t find it) about a health department who conducted a point of distribution (POD) exercise during the holidays.  Instead of handing out Tic Tacs or some other silliness, they did something great for their community – they distributed turkey dinners to those in need.  As I don’t have the article to reference and I had only skimmed over it the first time through, I don’t have the details of how they pulled this off, but having participated in the planning of POD exercises (particularly those that have a direct impact on the community, such as one that distributed preparedness kits and information) I can surmise how they did it.

As most of my readers know, emergency management is a collaborative process.  While local health departments are responsible for medical points of distribution, they can’t do it alone.  These are massive efforts to inoculate or prophylax hundreds if not thousands of persons within a narrow time frame.  These efforts require cooperation and support from emergency management, law enforcement, fire service, EMS, hospitals, volunteer organizations, and the private sector.  Commodity PODs can also be established, not necessarily run by the health department, with the intent of distributing needed commodities – such as tarps, food, or water – to the populace.  Health departments, however, are required to exercise their POD plans, which requires registration, intake, education, and inoculation of citizens.

In the example I linked to in the first paragraph regarding the preparedness kits, the health department was able to purchase most items and utilized a mix of staff and volunteers to run the POD, with support from other agencies to address traffic, parking, and other needs.  In the turkey dinner exercise, I imagine they were able to pay for some items and had others donated for this worthwhile effort.  It’s a great way of supporting the community with an immediate need while preparing for a future need.  Kudos to that community!

Now if I can only find that article…

~~~~~

6/17/14 Edit… I found a reference!

LLIS posted an ‘Innovative Practice’ bulletin about this exercise.  It can be found here.  To clarify/correct, it was actually an SNS exercise.

 

© 2014 Timothy Riecker

Verizon Ready for Storm Season

Great article from the Wall Street Journal on Verizon’s preparedness efforts for hurricane season.  The article seems to indicate all the right preparations, from planning and exercises to equipment and staging, response teams, and their own corporate EOC.  Much of our critical infrastructure is owned and operated by the private sector – it’s good to see the measures of preparedness Verizon takes responsibility for.  It’s certainly a model for other companies and industries – and even local governments.

-TR

Creating Operational Emergency Plans

I was inspired for this article from an email I received earlier today from Lu Canton, a rather prolific emergency management consultant who has branched a bit into consulting consultants.  His email today (a forward from his blog) was about making emergency plans ‘real’.  His point was that many planners focus on checking the boxes of the list of planning requirements (those prescribed by law, regulation, etc.) rather than focusing on ensuring that you have a plan that can actually be implemented.  He conducted a webinar over a year ago which I had blogged about.

Planning requirements are important, as they largely stem from lessons learned from earlier incidents.  Granted, some of these requirements come about being translated through the eyes and ears of politicians whose staffers write the legislation and don’t understand emergency management at all – resulting in convoluted, contradictory, and poorly focused requirements.  Requirements lead to standards, helping to ensure that emergency managers are addressing the needs of their jurisdiction and best practices in the industry.  To help guide us through this, many higher level agencies provide templates.  I’ve pontificated in the past about the danger of templates, which have a place in reminding us of these requirements and help us with format and flow, but are often misused by individuals who simply seek to fill in the blank with the name of the jurisdiction and claim they have a finalized plan.

How do we avoid falling into this trap?  Follow the planning process!  FEMA’s Comprehensive Preparedness Guide (CPG) 101 provides an overview of this process for creating emergency operations plans.  The two initial steps – forming a planning team and conducting a hazard analysis – are absolutely critical to the integrity of the process and ensuring a quality plan that meets the needs of your jurisdiction by addressing your threats and hazards.  Planning teams then need to consider these threats and hazards, make reasonable assumptions about their impacts (using a credible worst case scenario), then identify resources and strategies the jurisdiction will undertake to solve the problems they will face.

Does all this mean that a plan needs to be written from scratch?  Of course not!  In fact I strongly encourage people against it.  It’s practically guaranteed that you will forget a critical element.  One of the greatest things in the emergency management community is how we learn from each other.  You can reference templates you find, examine plans of your neighboring jurisdictions or jurisdictions similar to you, check out what is on LLIS.  There is plenty of great content you can examine and apply for your own use.  Just ensure that you carefully review and consider how it applies to you.

As you write the plan, think the details through.  This will help ensure that your plan is operational, not just meeting requirements.  Discuss with your planning team what is expected of each assisting and cooperating agency for each incident type.  Who will be in charge?  What resources will be necessary and where will you get them from?  What would the objectives be and what processes and decision points must be conducted to accomplish those objectives.  As you create the plan, map out these processes and ensure that you’ve considered the who, what, where, when, and how of each step in each process.  Recall that you are planning at a strategic level, not a tactical level.  Planning at a tactical level is nearly impossible with a pre-incident (aka ‘deliberate’) plan.  Tactics will be addressed during the actual response, hopefully referencing the EOP/CEMP you are writing now, and implemented through an incident action plan (IAP).

Remember, though, the proof is in the pudding, as they say.  Your plan needs to be tested to ensure viability.  Use a table top exercise to test policy and decisions, then a functional exercise to test the implementation of the plan and higher level tactics.  Full scale exercises and drills can test the tactical implementation of plans.  Good evaluation of the exercises will lead to planning improvements.  For insight on the exercise process, you can check out my exercise management series of posts referenced here.

Remember: when it comes to planning – keep it real!

Tim Riecker

Critical Infrastructure Vulnerability – Water Delivery Systems

Picture of the Baltimore Water Main Break, July 18, 2001

40-inch Water Main Break, Baltimore July 18, 2001. Source: Baltimore Sun

I’ve written several posts in the past on the vulnerabilities of our electrical infrastructure – both to natural and human causes.  Yet, our electrical infrastructure is not the only element of critical infrastructure that is vulnerable to failures and attacks.   We have a very old water infrastructure in our nation, with many areas still maintaining Civil War era cast iron pipes, with an estimated useful life of 150 years (at the time of installation).  How often does your area experience a water main break?

According to the US Conference of Mayors A major symptom of the aging water infrastructure includes 300,000 water main breaks in North America as result of the widespread corrosion problems adding up to a $50.7 billion annual drain on our economy. Leaking pipes are also losing an estimated 2.6 trillion gallons of treated drinking water annually (17 percent of all pumped water in the US), representing $4.1 billion in wasted electricity every year.”

This aging infrastructure has also failed us when we needed it most.  You might recall the Howard Street Rail Tunnel fire in Baltimore, MD on July 18, 2001.  I’ve designed exercise scenarios based upon this incident and have even received feedback from participants about the scenario being unlikely – they are rather surprised to learn that it is based on an actual event.

From the USFA Report on the incident:

“At 3:07 p.m. on Wednesday, July 18, 2001, a CSX Transportation train derailed in the Howard Street Tunnel under the streets of Baltimore, Maryland. Complicating the scenario was the subsequent rupture in a 40-inch water main      that ran directly above the tunnel. The flooding hampered extinguishing efforts, collapsed several city streets, knocked out electricity to about 1,200 Baltimore Gas and Electric customers, and flooded nearby buildings. The crash interrupted a major line associated with the Internet and an MCI WorldCom fiber optic telephone cable.

Throughout the incident, fire officials were plagued with three problems: fighting the fires in the tunnel; the presence of hazardous materials; and the weakening structural integrity of the tunnel and immediate surrounding areas.”

In reading the report you will see that the water main break both help and hurt the response.  The 40-inch main flooded streets and nearby businesses, but also was allowed to flow into the tunnel for a period of two hours, helping to decrease the temperature in the tunnel.  While no reports seem to indicate the impact of the water main break on nearby hydrants, I do include that impact in my exercise scenarios.

Water main breaks plague many areas around the nation.  The lack of potable water resulting from them creates a public health concern, resulting in many businesses and public buildings shutting down and households advised to boil water.  These breaks impact our ability to fight fires and, as a result of undermining, they can cause sink holes and damage to roadways.

In speaking to public works officials through the years, I’ve been told that every water system has leaks of varying severity.  Minor leaks often go undetected for a great period of time.

 

Securing our water supply is important as well.  Much of our water storage is in reservoirs, open and vulnerable to intentional contamination.  Most reservoirs have some measure of passive security (fences) and some even take more active security precautions.  However, we know that people who are determined can overcome these systems.  Luckily the sheer volume of water in most reservoirs would severely dilute any contamination introduced to them, but there may be agents so concentrated as to inflict harm.  The City of New York, for example, has a massive water supply system, with reservoirs as far north as the Catskills.  Their aqueducts, made famous in the third Die Hard movie, are massive.  The New York City Department of Environmental Protection is charged with securing the City’s water supply and does so through both active and passive security measures as well as active and on-demand water quality sampling.  Most areas, however, don’t have these law enforcement or public health resources available in such abundance.

Water is a critical component of our infrastructure and therefore must be protected.  It’s important not only to business and industry, but is also essential to human life, agriculture, and food production.  Similar to our roadways and electrical infrastructure, our water systems need a plan for restoration and funding to put that plan into action.  Beyond some more capable and financially stable municipalities, most water systems are implementing ad-hoc fixes and are only able to replace small sections of the system each year.

Does your plan account for water system failure?

Tim Riecker