EMPG ROI 2022: Another Wasted Effort

The National Emergency Management Association (NEMA) and the International Association of Emergency Managers (IAEM), the two most prominent membership organizations in the US for emergency managers, one again released their joint annual report: Emergency Management Performance Grant: An Evaluation of the Nation’s Return on Investment. I touched on last year’s report in a post I made about metrics and data analysis. Ironically, a couple months prior to that post, I wrote about measuring return on investment through the use of key performance indicators (KPI).

Clearly some of the people who SHOULD have read these didn’t. This year’s report on EMPG return on investment is pretty much the same as last year’s, simply with updated numbers. To call the content of the report an evaluation of return on investment of this important grant program is a considerable overstatement and does nothing to support emergency management. The numbers, such as x number of people trained or how much money was spent on plan development, are largely superficial and don’t really provide any analysis of return on investment. As mentioned in the articles I authored last year which are referenced above, we should be reporting on key performance indicators and drilling down to identify what needs have been met through the efforts and investments. Included in the report are a few anecdotes of ‘EMPG-Supported Success’ that tell more of a story and provide more valuable information than the scant bit of statistical analysis. But really, this report doesn’t tell me anything. It provides little to no benefit to state and local emergency managers, which are the majority membership of both organizations. As a dues-paying member of NEMA, I’m disappointed in this effort and expect better from them. As an emergency manager I continue to be frustrated that we, as a professional practice, continue to accept this kind of information and reporting. Let’s raise our expectations and demand better.

© 2022 Tim Riecker, CEDP

Emergency Preparedness Solutions, LLC®

Measuring Return on Investment Through Key Performance Indicators

Return on investment (ROI) is generally defined as a measurement of performance to evaluate the value of investments of time, money, and effort. Many aspects of preparedness in emergency management offer challenges when trying to gauge return on investment. Sure, it’s easy to identify that m number of classes were conducted and n number of people were trained, that x number of exercises were conducted with y number of participants, that z number of plans were written, or even that certain equipment was purchased. While those tell us about activity, they don’t tell us about performance, results, or outcomes.

More classes were conducted. So what?

We purchased a generator. So what?

The metrics of these activities are easy to obtain, but these are rather superficial and generally less than meaningful. So how can we obtain a meaningful measure of ROI in emergency preparedness?

ROI is determined differently based on the industry being studied, but fundamentally it comes down to identifying key performance indicators, their value, and how much progress was made toward those key performance indicators. So what are our key performance indicators in preparedness?

FEMA has recently began linking key performance indicators to the THIRA. The Threat and Hazard Identification and Risk Assessment, when done well, gives us quantifiable and qualifiable information on the threats and hazards we face and, based upon certain scenarios, the performance measures need to attain certain goals. This is contextualized and standardized through defined Core Capabilities. When we compare our current capabilities to those needed to meet the identified goals (called capability targets in the THIRA and SPR), we are able to better define the factors that contribute to the gap. The gap is described in terms of capability elements – planning, organizing, equipping, training, and exercises (POETE). In accordance with this, FEMA is now making a more focused effort to collect data on how we are meeting capability targets, which helps us to better identify return on investment.

2021 Emergency Management Performance Grant (EMPG) funding is requiring the collection of data as part of the grant application and progress reports to support their ability to measure program effectiveness and investment impacts. They are collecting this information through the EMPG Work Plan. This spreadsheet goes a long way toward helping us better measure preparedness. This Work Plan leads programs to identify for every funded activity:

  • The need addressed
  • What is expected to be accomplished
  • What the expected impact will be
  • Identification of associated mission areas and Core Capabilities
  • Performance goals and milestones
  • Some of the basic quantitative data I mentioned above

This is a good start, but I’d like to see it go further. They should still be prompting EMPG recipients to directly identify what was actually improved and how. What has the development of a new plan accomplished? What capabilities did a certain training program improve? What areas for improvement were identified from an exercise, what is the corresponding improvement plan, and how will capabilities be improved as a result? The way to get to something more meaningful is to continue asking ‘so what?’ until you come to an answer that really identifies meaningful accomplishments.

EMPG aside, I encourage all emergency management programs to identify their key performance indicators. This is a much more results-oriented approach to managing your program, keeping the program focused on accomplishing meaningful outcomes, not just generating activity. It’s more impactful to report on what was accomplished than what was done. It also gives us more meaningful information to analyze across multiple periods. This type of information isn’t just better for grant reports, but also for your local budgets and even routine reports to upper management and elected officials.

What do you think about FEMA’s new approach with EMPG? What key performance indicators do you use for your programs?

© 2021 Timothy Riecker, CEDP

Emergency Preparedness Solutions, LLC®