EMPG ROI 2022: Another Wasted Effort

The National Emergency Management Association (NEMA) and the International Association of Emergency Managers (IAEM), the two most prominent membership organizations in the US for emergency managers, one again released their joint annual report: Emergency Management Performance Grant: An Evaluation of the Nation’s Return on Investment. I touched on last year’s report in a post I made about metrics and data analysis. Ironically, a couple months prior to that post, I wrote about measuring return on investment through the use of key performance indicators (KPI).

Clearly some of the people who SHOULD have read these didn’t. This year’s report on EMPG return on investment is pretty much the same as last year’s, simply with updated numbers. To call the content of the report an evaluation of return on investment of this important grant program is a considerable overstatement and does nothing to support emergency management. The numbers, such as x number of people trained or how much money was spent on plan development, are largely superficial and don’t really provide any analysis of return on investment. As mentioned in the articles I authored last year which are referenced above, we should be reporting on key performance indicators and drilling down to identify what needs have been met through the efforts and investments. Included in the report are a few anecdotes of ‘EMPG-Supported Success’ that tell more of a story and provide more valuable information than the scant bit of statistical analysis. But really, this report doesn’t tell me anything. It provides little to no benefit to state and local emergency managers, which are the majority membership of both organizations. As a dues-paying member of NEMA, I’m disappointed in this effort and expect better from them. As an emergency manager I continue to be frustrated that we, as a professional practice, continue to accept this kind of information and reporting. Let’s raise our expectations and demand better.

© 2022 Tim Riecker, CEDP

Emergency Preparedness Solutions, LLC®

Measuring Return on Investment Through Key Performance Indicators

Return on investment (ROI) is generally defined as a measurement of performance to evaluate the value of investments of time, money, and effort. Many aspects of preparedness in emergency management offer challenges when trying to gauge return on investment. Sure, it’s easy to identify that m number of classes were conducted and n number of people were trained, that x number of exercises were conducted with y number of participants, that z number of plans were written, or even that certain equipment was purchased. While those tell us about activity, they don’t tell us about performance, results, or outcomes.

More classes were conducted. So what?

We purchased a generator. So what?

The metrics of these activities are easy to obtain, but these are rather superficial and generally less than meaningful. So how can we obtain a meaningful measure of ROI in emergency preparedness?

ROI is determined differently based on the industry being studied, but fundamentally it comes down to identifying key performance indicators, their value, and how much progress was made toward those key performance indicators. So what are our key performance indicators in preparedness?

FEMA has recently began linking key performance indicators to the THIRA. The Threat and Hazard Identification and Risk Assessment, when done well, gives us quantifiable and qualifiable information on the threats and hazards we face and, based upon certain scenarios, the performance measures need to attain certain goals. This is contextualized and standardized through defined Core Capabilities. When we compare our current capabilities to those needed to meet the identified goals (called capability targets in the THIRA and SPR), we are able to better define the factors that contribute to the gap. The gap is described in terms of capability elements – planning, organizing, equipping, training, and exercises (POETE). In accordance with this, FEMA is now making a more focused effort to collect data on how we are meeting capability targets, which helps us to better identify return on investment.

2021 Emergency Management Performance Grant (EMPG) funding is requiring the collection of data as part of the grant application and progress reports to support their ability to measure program effectiveness and investment impacts. They are collecting this information through the EMPG Work Plan. This spreadsheet goes a long way toward helping us better measure preparedness. This Work Plan leads programs to identify for every funded activity:

  • The need addressed
  • What is expected to be accomplished
  • What the expected impact will be
  • Identification of associated mission areas and Core Capabilities
  • Performance goals and milestones
  • Some of the basic quantitative data I mentioned above

This is a good start, but I’d like to see it go further. They should still be prompting EMPG recipients to directly identify what was actually improved and how. What has the development of a new plan accomplished? What capabilities did a certain training program improve? What areas for improvement were identified from an exercise, what is the corresponding improvement plan, and how will capabilities be improved as a result? The way to get to something more meaningful is to continue asking ‘so what?’ until you come to an answer that really identifies meaningful accomplishments.

EMPG aside, I encourage all emergency management programs to identify their key performance indicators. This is a much more results-oriented approach to managing your program, keeping the program focused on accomplishing meaningful outcomes, not just generating activity. It’s more impactful to report on what was accomplished than what was done. It also gives us more meaningful information to analyze across multiple periods. This type of information isn’t just better for grant reports, but also for your local budgets and even routine reports to upper management and elected officials.

What do you think about FEMA’s new approach with EMPG? What key performance indicators do you use for your programs?

© 2021 Timothy Riecker, CEDP

Emergency Preparedness Solutions, LLC®

Gauging Return on Investment in Preparedness: Training

This is my fourth article in a series examining how organizations can gauge their return on investment for various emergency management and homeland security preparedness projects.  These were inspired by an original article I wrote called Measuring Return on Investment in Emergency Management and Homeland Security: Improving State Preparedness Reports.  The POETE model (Planning, Organizing, Equipping, Training, Exercising) helps us to identify all activities related to preparedness.  Thus far, we have covered:

Planning

Organizing

Equipping

If you haven’t already reviewed these articles, check them out for additional context and information.  Within each, I outline the general activities within the preparedness element, and identify the potential costs and benefits of these activities to the organization.  While some costs and benefits are direct (meaning we can readily identify them in terms of currency), most are not, and require some measure of analysis.

We put a lot of money (and faith) into training as a central preparedness activity.  And why shouldn’t we?  Training, by definition, is a transfer of learning.  We have a lot of information to communicate to our staff and other stakeholders, with the goal of that information collectively becoming a body of knowledge, and a vision of these people applying what they have learned to future circumstances.

In emergency management and homeland security we train quite a bit, with some training being required by organizational, local, state, national, international, or federal standards; while other training will help develop and advance staff.  We include HR-required training; soft skills like communication, leadership, decision making, and the like; as well as technical skills such as emergency planning, exercise design, and incident management.  Emergency management and homeland security are broad fields of practice, which intersect public safety, public health, and other essential government and social functions.  Most emergency management and homeland security practitioners have roots in one or more of these fields and typically continue receiving training relative to those as well.

Training comes from a variety of sources including our home organizations, local and county governments, state government, private and not for profit entities, and the federal government.  Particularly for training that is sponsored by a government entity, most training is ‘free’.  In the US, federal training entities, such as FEMA’s Emergency Management Institute, also reimburse travel expenses and provide lodging for all levels of government employees.  On the surface, the cost of most emergency management and homeland security training is fairly low.

Of course depending on perspective, the cost of training can vary.  As a former state training officer who managed all emergency management related training delivered across my state, I could identify our agency’s cost of training.  This would include our staff admin time for course prep and record keeping, the cost of duplicating participant manuals, any costs associated with the hosting facility (although we usually utilized no cost facilities), the cost of paying our instructors for their time and travel, and, if applicable, any lodging and/or meal costs for participants who may have traveled a distance.  These costs, however, are only part of the picture.

What about the cost to the organization that is sending people to be trained?  Directly, this is salary time in which little to no work is actually being accomplished for the organization.  The organization may also be footing the bill for travel costs for their participants.  Depending on who is sponsoring the training, there may be a fee for the course.  Indirectly, what is the cost of that employee being away?  Who is doing their work?  This often depends on the organization and the position the individual holds in that organization.  Firefighters, police officers, health care professionals, and others may be covering shifts that will still need to be filled, especially if policies or union contracts require certain staffing levels.  Sometimes this backfilling isn’t as simple as changing schedules, as most employees are already scheduled at full time status.  Therefore, someone may need to be paid overtime to fill this position for the duration of the training.  Perhaps the learner themselves is being paid overtime to tend to priority tasks outside of training hours.  Maybe others can simply absorb some extra tasks while this person is gone, or the learner will be swamped for some period of time once they return from training.  Each of these mechanisms, all dealing with productivity, has a cost associated to it.

Training can get expensive, which is why it’s often one of the first activities cut when an organization’s budget gets tight.  We try to minimize the cost of training through a variety of practices such as shorter training days, online training, and local training.  These, however, have various impacts on the organization’s ability to obtain the training as well as the overall effectiveness of the training.  Sometimes we simply defer the training, but the organization may have little choice, particularly with mandated training.

So what benefits can training provide?  As mentioned, I have quite a bit of background as a trainer and training manager.  I’d love to tell you that training has the greatest of all benefits, but that would be a complete lie. Just like any other preparedness activity, it has to be properly applied.  Training won’t fix everything.  I’ve written a lot of pieces on training in my blog… just search for ‘training’ and you will find plenty of articles about how training should/shouldn’t be applied.  With that caveat, training can have great benefit.  Not only can it make people more effective and efficient in their jobs and related tasks, it can also help defer liability from the organization.  Training also has benefits which more directly apply to the learner vs the organization, such as providing background for advancement and/or promotion (which can be internal or external to the organization).  There are many practices in emergency management and homeland security for which training aids in health and safety, not only of the staff member who took the training, but also of others.  In training, the impacts can go pretty far… you just have to follow the bouncing ball.  As an example: Jane gets trained in how to write emergency plans.  The emergency plans she writes will help the organization respond more effectively in the event of a disaster.  When the organization responds more effectively, lives and property are protected.  While this is a great ideal outcome, it makes for some difficulty in determining the benefits in terms of currency.

Often, the most direct benefits from training are rooted in compliance and proficiency.  Organizations have a variety of compliance matters they have to meet.  These obligations may be HR driven, required by an executive, a higher level of government, an accreditation body, or a funding source.  Safety matters are also usually linked with a compliance matter.  I often try to associate training activities to these requirements.  Sometimes we can directly link a financial benefit to these compliance matters, while other times compliance is simply factually stated.  Second is proficiency.  People need to stay current in essential skills.  This might require regularly recurring training for staff, well as training for new staff.  Staff need to be proficient in new procedures, software, and equipment operation.  Certain staff may need to be trained to more advanced levels.  Gauging the benefit in financial terms for proficiency is generally more difficult, although the need for the training is apparent.  The benefit simply needs to be extrapolated.  For instance, if the training is in a new process, what is the time and/or quality difference between the old process and the new?

As mentioned earlier, it is often times not easy to determine the financial return on investment for many of our activities.  We need to dig deep and identify quantifiable metrics which can be examined before and after we apply our preparedness activity.  We must assign reasonable currency figures to those metrics to help us and other decision makers better understand our investment and the benefits it will potentially bring.

Soon I’ll be wrapping up this series with the last key preparedness activity – exercises.  As always I’m happy to hear your thoughts on how we can better identify the returns on our investments in preparedness activities.  As a resource, I’d encourage you to search Training Magazine (trainingmag.com), which often has articles on analyzing the return on investment in training.

© 2015 – Timothy Riecker

Emergency Preparedness Solutions, LLC

Gauging Return on Investment in Preparedness: Organizing

As a continuation of the Gauging Return on Investment in Preparedness series (read the first one on Planning here), this post’s focus is on the POETE element of Organizing.  There are a number of ‘organizing’ efforts we engage in through our preparedness endeavors.  Some are temporary, like establishing working groups to solve a certain problem; while some are intended to be long-term, like forming an incident response team.

Why organize?  Most organizational efforts are fueled by the need to capitalize on the power of many.  What one person can do, more people can do better.  Problem solving, responding, etc.  Often our organizational efforts are internal, but, particularly in public safety, we coordinate with other agencies.  We might be building a professional response organization, such as an Incident Management Team (IMT), or perhaps we are building a community organization, such as a Community Emergency Response Team (CERT).

What costs are associated with organizational activities?  Foundationally, it’s simply the staff time needed to prepare for, attend, and perform follow up work from meetings and other organizational efforts.  Depending on how complex our efforts are, however, and the intent of our organizational efforts, this can take on full time duties.  You also have to consider who is being drawn into these efforts and what the ‘replacement cost’ is of their time – meaning, what is the cost of someone else performing their work while they are involved in the meetings, etc.?  We also need to identify what costs might be associated with organizing?  The remaining POETE elements (Planning, Equipping, Training, and Exercising) can probably lead you to identifying these.

What are the benefits (value) of organizing?In order to identify the return on our investment, we need to be able to ascertain the benefits our organizational efforts bring – some may be tangible and relatable in dollar figures, others may be more intangible and amorphic.

As with many preparedness efforts, we find ourselves needing to make reasonable assumptions to identify cost savings or value.  We need to follow the bouncing ball of our efforts.  As an example… If we create a CERT team, citizens will be better able to tend to their own needs in the event of a disaster.  This leads to less immediate need of limited resources (first responders), allowing them to focus on more critical needs (i.e. saving lives and protecting infrastructure).   In this example we can make some assumptions about the types of infrastructure to be impacted by a certain incident and the costs associated with it becoming incapacitated.

In regard to saving lives, it’s difficult for us to attach a dollar value to that.  We often say that lives are priceless, and while that may be true, we sometimes need to make an educated guess.  Depending on who you are reporting figures to, they may be satisfied with a reasonable number of lives being saved… others may want to actually compare apples to apples (that is, dollars to dollars).  If you engage the use of your favorite internet search engine and search ‘what is the value of a life’, or something similar, you will find a number of results.  In perusing some of these results myself, I found that the dollar figure assigned to a life is obviously subjective and very much related to the industry in which the question is being asked.  This particular article makes for an interesting read on the subject.  Spoiler alert: they peg the value of a human life at $5M USD (2011).

In the end, organizational efforts need to have a purpose providing a net value.  Even in routine matters and daily business, we should examine the cost of organizational efforts – particularly meetings.  Meetings are one of my biggest bugaboos, as they are often too long, have little purpose, and the objectives can be met in a much more efficient manner.

What ideas do you have on determining the return on investment for organizational activities?

© 2015 – Timothy Riecker

Emergency Preparedness Solutions, LLC

www.epsllc.biz 

Gauging Return on Investment in Preparedness: Planning

Inspired a bit by my previous post Measuring Return on Investment in Emergency Management and Homeland Security: Improving State Preparedness Reports, I’ve decided upon writing a series of posts picking apart our primary activities in emergency management and homeland security preparedness to identify ways to gauge our Return on Investment (ROI).  To encapsulate our primary activities, I’m using the five POETE capability elements:

  • Planning
  • Organizing
  • Equipping
  • Training
  • Exercises

Most preparedness activities within emergency management and homeland security fall within one or more of the POETE capability elements.  The capability element of Planning is the foundational activity on which all preparedness is built and will be the topic of this post.  Here’s what I’m covering:

  • What is Return on Investment?
  • What planning efforts are involved in preparedness?
  • What organizational investments are involved in Planning?
  • Does the planning effort comply with applicable standards?
  • Can the plan be implemented?
  • What will exercises tell you?
  • Is there a need to maintain plans?

Return on Investment, or ROI, is a business term used to identify the profitability of certain investments or actions.  While preparedness is certainly done to protect against losses, for public and private sector alike, we generally don’t see preparedness activities as generating revenue.  However, when most entities INVEST time, money, and other resources into preparedness activities, they often want a reasonable assurance that their investment has paid off.  How do we gauge ROI for planning efforts?

First off, what planning efforts might we see in public or private organizations?  Obviously emergency and disaster plans are the big ones.  These plans are designed to identify key processes, such as alert and notification, response organization and incident management, and others which are intended to save lives and protect property.  These plans are likely to have annexes and appendices which address uniqueness of certain hazards, response circumstances, and support activities.  Continuity plans – usually business continuity or government continuity – identify how the organization will survive as an entity in the face of disaster.  Planning activities also involve the creation, review, and maintenance of policies and procedures.  We also create plans for hazard mitigation, long term recovery, specific events, and other needs.

What investments are involved in planning activities?  Organizations can and should allocate staff time and physical space and infrastructure to planning efforts.  The dedication of staff (full or part time) and/or consultants is often required, especially when planning efforts are viewed as a continual process and a critical part of preparedness.  The organization itself must make a commitment to the planning effort.  This commitment isn’t just in concept, but also practical involvement of staff throughout the organization, access to information, and even an involvement of third parties.

Certainly a first step in assessing return on investment of planning is to evaluate compliance with applicable rules, regulations, and guidelines.  These requirements can be hard (legally binding) or soft (general guidance) and can differ from industry to industry, nation to nation, and state to state.  Here in the US, FEMA provides guidance on emergency planning through Comprehensive Preparedness Guide (CPG) 101.  Some states may have requirements for emergency planning, such as New York State’s Executive Law Article 2-bNFPA 1600: The Standard on Disaster/Emergency Management and Business Continuity Programs is often referenced by public and private entities alike, while the International Standards Organization (ISO), has many industry-specific requirements for emergency planning.  If grant funding is being used for the planning effort, the grant may also have specific requirements.  Regardless of what the requirements are, planning efforts, plans, and associated documents should be audited to ensure that requirements are met.

Compliance, however, isn’t necessarily indicative of a good planning effort.  I’ve seen many plans which may meet requirements but the content itself was severely lacking.  Far too often planners get caught up in the world of checking boxes and fail to consider implementation.  If a plan cannot be implemented, it is useless to the organization.  Many plans exist now that meet applicable requirements but are still yet vacant of any meaningful direction or guidance in the event of an emergency.  These types of ‘plans’ are really better seen as policy documents.  A plan should identify what will be done, when, how, and by who.  If your ‘plan’ simply contains a statement on the requirement to use NIMS/ICS, but doesn’t provide detail on who will be in charge of what, when, and how; it is a policy document, not a plan.  Plans and their associated documents (i.e. procedures, guidelines, and job aids) need to chase down the lifespan of each critical step, especially early in a response.  They must identify who is responsible to make key decisions, who will be notified (how and by who), and who will take what actions.  A logical review of planning documents by the planning committee or perhaps even a third party is another good means of assessing your return on investment.

Does the plan work?  This is, perhaps, the ultimate factor in determining return on investment.  Usually our best means for identifying if a plan works is to exercise it.  Exercises provide a controlled and focused environment for testing plans or components of plans.  They will also help us in identifying if the plan can truly be implemented.  I’ve written a lot on exercises: articles can be found here.  (I also anticipate writing about assessing ROI for exercises as part of this series).  Generally, an incremental exercise program is usually recommended, beginning with discussion-based exercises – such as table tops and workshops – and progressing to operations-based (hands on) exercises.  A well written and honestly evaluated exercise will go a long way toward identifying the return on investment of your planning efforts.

Are we there yet?  Nope.  Planning, like all other preparedness efforts, requires maintenance.  If you create a plan then walk away, even if it’s a good plan, your plan’s value will diminish over time – and we’re talking months, not decades.  Think about how often something changes in your organization.  Staffing.  Equipment.  Technology.  Procedures.  Insurance policies.  All of these things, and more, influence your plans in some way.  Over time these changes not only occur, but also compound and move the present reality of your organization further from the assumptions of your planning efforts.  This is why plans must be maintained and updated on a regular basis.

Is there some mathematical formula for identifying the return on investment of preparedness efforts?  Given all the factors involved and their fluidity, I don’t think so.  It’s not cut and dry like a traditional business investment.  As you can see, though, there are a number of steps we can take to assess the utility of our investment.  I’ve seen organizations pay a lot for bad plans, and others pay much less for great plans.  Not only do organizations need to ensure that their planners know what they are doing, but the organization itself needs to have a commitment to success.  Without it, the planning effort is doomed to fail.

As always, feedback is appreciated.  What are your thoughts on assessing the return on investment of planning efforts?  What do you think is a good measure?

Does your organization need a new plan or need to update a plan?  Do you need help with the planning process or evaluating your organization’s preparedness?  How about exercises?  Emergency Preparedness Solutions can help!  Email to consultants@epsllc.biz or visit www.epsllc.biz.

© 2015 – Timothy Riecker

Emergency Preparedness Solutions, LLC

WWW.EPSLLC.BIZ

Measuring Return on Investment in Emergency Management and Homeland Security: Improving State Preparedness Reports

A lot of money is spent within the emergency management and homeland security enterprise.  Looking just at the last couple of years of annual Homeland Security Grant Program (HSGP) (this is the annual grant provided by the US Federal government to states and urban areas), $1.044 billion was allocated in FFY 2015 and $1.043 billion allocated in FFY 2014.  These billions of dollars only account for a portion of spending within EM/HS.  There are other federal initiatives as well as state, tribal, territorial, and locally funded efforts.  Businesses and NGOs also invest significantly in emergency management, homeland security, and business continuity activities.  But where does it all get us?

Through the past decade or so there have been a few efforts by DHS/FEMA to try to measure preparedness, ideally to identify improvements in our preparedness as the result of the billions of dollars invested.  None of these efforts have really provided obvious and tangible results.  The current measure is through annual State Preparedness Reports (SPRs), which utilize the THIRA process (Threat and Hazard Identification and Risk Assessment) outlined in Comprehensive Preparedness Guide (CPG) 201 as a foundation, but with the POETE analysis (Planning, Organizing, Equipping, Training, and Exercising) for each of the 31 Core Capabilities.  (You can find articles I’ve written on the utility and application of POETE here.) The SPR is a good methodology for identifying the current condition of preparedness for each state as viewed through the 31 Core Capabilities.  The POETE analysis helps to identify the strengths and weaknesses within each Core Capability.

The SPR, however, still falls short.  How can we improve it?

1. Include historical data for trend analysis. The SPR largely provides only a snapshot of current conditions.  The format of the SPR does not provide for any analysis of historical data to identify trends (i.e. improvements or otherwise) in the state’s assessed condition of its 31 Core Capabilities.  FEMA regional offices, upon receipt of an SPR, do provide a brief feedback report of the current SPR with a passing mention of the previous year’s submission, but the report provides so little information it could hardly be called an analysis.

A rudimentary table identifying trends for a selected Core Capability is below.  For those not familiar, higher scores rate a higher measure of capability.  In this table, I’ve identified POETE elements which have trended lower from year to year with a RED highlight, and those which have trended higher with a GREEN highlight.  A simple analysis such as this give an at-a-glance comparison.  To make this analysis more comprehensive, I would suggest the addition of narrative for each trend (higher or lower) which explains what has changed to warrant the new ranking.

Historical Comparison of a Core Capability

2. Include a financial analysis for the current year to identify return on investment. An identification and summary of key program area investments for the year will lay the groundwork for a return on investment (ROI) analysis.  ROI will help identify how much bang for the buck you are getting in certain areas.  It’s easy to lose sight through the year from a program management perspective on how much money was spent on certain programs and activities – especially for larger agencies with a layered bureaucracy.  Incorporating this analysis into an SPR is not only good financial and program management, but provides an opportunity to identify where money was spent and to measure, at least on a broader scale, what the results were.  Certainly we have to fund continued operations to simply sustain our capabilities, but we should also be funding, where possible, programs to enhance our high priority capabilities and those needing the most improvement.

Again, as a rudimentary example, we can build on the table provided earlier to identify where funds were spent to see if they made a difference in our level of preparedness.  As with the earlier example, a narrative should be provided for each investment to identify what it was and assess the impact.  This also provides an excellent opportunity to review the investment justification written for grants to determine if the investment met the intended objectives (which should have been to maintain or enhance some aspect of the capability).  Historic investment data can also be included for each year.  This all leads directly to identifying the return on investment – did the investment make a difference and to what extent?

Historical Analysis of a Core Capability with Identified Investments


Ultimately this added data and analysis requires more work, potentially the involvement of more people, and likely more time to complete the SPR.  However, this new process will also result in a positive return on investment itself by helping to identify trends and outcomes.  Financial information is regularly reported to DHS (for those grants that originate with them) in the form of progress reports, but that information is stovepiped and usually not associated with a more comprehensive assessment such as the THIRA/SPR.  Bringing this data together paints a much more accurate picture.

The concept of preparedness is difficult to put in a box.  It’s amorphic and challenging to identify, yet people often ask the question ‘Are we prepared?’  States, locals, DHS, and Congress often have difficulties measuring preparedness and advances in preparedness, especially relative to the dollars spent on it.  The GAO has regularly recommended efforts to better identify return on investment, yet we haven’t gotten there.  The recommendations identified here can bring us much closer to nailing down where we are and where we need to be.  Armed with this knowledge, we can make better decisions for future investments and activities.

Moving forward, I expect to write a bit on each POETE element, with my thoughts on how we can identify return on investment for each.  As always, I’m very much interested in your thoughts on the approach I identified above and how we can better identify return on investment in the realm of emergency management and homeland security.

If you are interested in utilizing this approach to better identify your return on investment for local, state, tribal, territorial, or organizational preparedness efforts (whether or not you do a State Preparedness Report), Emergency Preparedness Solutions is here to help!  Check out our website at www.epsllc.biz or contact me directly to discuss what we can do for you.

© 2015 – Timothy Riecker

Emergency Preparedness Solutions, LLC

WWW.EPSLLC.BIZ