Assuring fair, respectful and motivating evaluation
The performance evaluation of Next-Generation Emergency Services (NGES) is based on the NEXES Flexible Key Performance Indicators (Flexible KPIs). These Flexible KPIs have been engineered for a socio-technical evaluation of NGES performance, fitting to the emergency services’ major business processes while being customisable for each individual emergency service organisation. The structured approach supports both one-time evaluation and frequent (re)evaluations of current and historical data suitable for monitoring and (re)planning).
The evaluation of NGES should be objective and repeated at key stages following adoption; going beyond traditional quantitative, one-dimensional KPIs. The evaluation should include both quantitative and qualitative effects of NGES. Given the differences among Emergency Services, the evaluation should be agreed upon and standardised to apply to all Emergency Services while at the same time being customisable to the individual situation of each Emergency Service organisation.
The evaluation should be explainable in terms of its methods, parameters and priorities used, to enhance transparency, evaluation and comparison within an Emergency Service organisation over time, as well as with other Emergency Services.
Within the NEXES Action, we decoupled the single effect measured by a (traditional) KPI from the definition of the KPI, leading to Flexible KPIs. These Flexible KPIs are contextualised specifically for the emergency services’ context as NEXES Flexible KPIs. First, a set of 11 KPIs is defined and grouped into categories ranging from ‘Emergency Calls’, ‘Defining the Situation’, ‘Coordination of Response Activities’, and ‘Communication to Citizens’, to ‘Customer Satisfaction’, where each KPI is mandatory. That means each emergency service organisation must have a score for all eleven KPIs. A KPI is not defined as a single effect, rather each Flexible KPI is dependent on multiple effect measurements. Each Flexible KPI can draw on the appropriate and relevant effect measurement subset in order to tailor the evaluation via the KPI to the specific organizational context. The overall score of a KPI can be calculated by aggregating the scores for the relevant and tailored effect measurements.
NEXES Flexible KPIs categorised in the main business processes of emergency services.
As shown in figure above, the Flexible KPIs are categorised according to the four major business processes of an emergency service organisation. The fifth category, customer satisfaction, is added to include the perspective of both ES professionals and citizens into the overall evaluation of NGES.
Support autonomy
The approach chosen by NEXES is thus to support the autonomy of Emergency Service organisations. Here, ‘autonomy’ refers to the ability of an emergency service organisation to govern itself, make its own plans, and decide when and how to adopt specific NGES solutions. Any approach that limits the autonomy of emergency services is likely to encounter significant problems in implementation.
The autonomy of emergency services is supported in two ways. First, by requiring an emergency service organisation to select those effect measurements that are relevant to their situation. Second, by allowing the emergency service organisation to tailor the relevant effect measurements to their specific situation. Furthermore, NEXES requires that all of these decisions and adaptations are made visible. Having these aspects visible makes it possible to compare emergency services’ adoption of NEXES solutions; the original reason for introducing Key Performance Indicators.
Within NEXES, an effect measurement has a structured description. An emergency service organisation is invited to ‘fill in the blanks’, thereby tailoring the effect measurement to their situation. For example, consider effect measurement #31, which measures ‘Incident Location Accuracy’. This effect measurement measures the discrepancy between the incident location determined by the emergency services’ operators versus the actual location of the incident (which is objectively known after the incident has been mitigated). It is expected that using GPS-based location information from callers, the incident location becomes known more accurately than using the cellular base-stations triangulation algorithms, especially in more rural and elevated areas.
Furthermore, emergency services can specify how important certain effect measurements are when calculating their KPIs’ scores. The mechanism provided for this is that the score of an effect measurement gets an associated importance before aggregation into the KPI score. This is shown in the figure below, where the aggregation method makes it possible to ‘weigh the relative importance’ of factors contributing to KPI scores as well as the overall score on NGES impact.
Overview of the 11 KPIs in the 5 categories, all effect measurements are not shown.
For each emergency service organisation, we cannot ‘equalise’ the measurement methods: that would bring us back onto the “one size fits all” path. The measurement methods of the emergency services will be assumed to be different, although they may have similarities. Based on the resulting measured values, the evaluation by emergency services is tailored to their own needs. This tailoring is what we can re-use: we can use the tailoring of any one emergency service organisation and apply it to itself (e.g. after a change) or another emergency service organisation (e.g. out of curiosity). Thus we can compare the Flexible KPIs’ scores from the perspective of any emergency service organisation by applying tailoring to many emergency services, including the original emergency service organisation itself after a change. It also becomes possible to re-compute the scores for all emergency services, given new insights in e.g. the relative importance of specific measurement methods, or using similar reference values, or for any other compelling reason.
For the example emergency services, introduced in the previous article, each may have a different perspective on KPI priorities. For example, organisation A (rural) may place more emphasis on ‘Defining the Situation’ as it is quite common to have non-native speaking visitors in the region who cannot explain where they are and what particular problem they are facing. Meanwhile, organisation B (urban) may place more emphasis on ‘Communication To Citizens’, given e.g. possible hazardous situations with chemical substances in the harbour and surrounding industries and the desire to specifically address citizens based on location and/or other conditions.
These example organisations may also have different perspectives on relevance of effect measurements. For example, organisation A (rural) may consider effect measurements regarding video-calling and interactive-chat to be relevant given its current pilot with a special-purpose emergency-App for its citizens and tourists and the desire to further extend this app. Meanwhile, organisation B (urban) may consider these effect measurements of less relevance as there are currently no emergency app or other facilities in place.
Application of Flexible KPIs
The flexibility of the KPIs and effect measurements has been shown to be beneficial (1) to guarantee autonomy of emergency services in the application and tailoring of the KPIs, (2) to enable structured comparison of results from the before and after situations of an emergency service organisation, and (3) to foster exploration and understanding of the NEXES impact by changing and re-computing scoring formula and scores of KPIs for (individual / a set of /all) emergency services.
Application of the Flexible KPIs usually follows a number of steps:
 |
First and foremost the emergency service organization must decide what it is necessary to measure, also given objectives and policies provided by policy-makers and decision-makers: data collection always takes time and effort. This includes making the trade-off regarding which measurements can be collected continuously and which measurements are collected frequently or after certain events. |
|---|---|
 |
Second, priorities and relevance must be determined for the various effect measurements and KPIs. Priorities can be changed for later re-evaluation. |
 |
Third, each effect measurement must be defined for this emergency service organisation: what is measured, how is the data collected, with what method is the data analysed, and with what method is the evaluation determined? |
 |
Fourth, the quantitative and qualitative data can be collected, analysed and evaluated as defined above. |
 |
Fifth: based on evaluation of each effect measurement, the KPIs can be computed. Changes in priorities and evaluations can easily be made and their significance can be studied. What-if studies are natively supported. |
An important advantage of the Flexible KPI tool is that it accommodates changes in data collection and evaluation thereof. It is thus possible to evaluate the KPIs at different moments in time, taking into account the different circumstances.
This article has described the NEXES Flexible KPIs suitable for evaluation of NGES performance. The next article describes how the Flexible KPIs can be used for both monitoring and planning activities.
This blog is number 2 in a series of five articles on the NEXES KPIs Lessons Learned and Recommendations. When you wish to delve deeper into the NEXES Action and its solution to comparing apples and oranges we recommend to read the deliverable D2.4, for more information on the NEXES Recommendations we recommend to read the deliverable D2.5. Below is the list of all the articles in the series:
- ‘Comparing apples and oranges’: The evaluation setting
- Flexible KPIs: Assuring fair, respectful and motivating evaluation
- Playing the ‘what-if’ game: Monitoring and planning with Flexible KPIs
- A public playing field: Pragmatic exploitation and availabaility
- Continuous Improvement: The Flexible KPIs as a tactical aid for emergency services
Dr. Niek Wijngaards works for AIMTech Consulting Limited in the United Kingdom and True Information Solutions in the Netherlands as senior consultant and solution architect. His focus on user-centered innovation and his work on intelligent systems and scenario-based robust decision-making provides a sound basis for the development of the NEXES Flexible KPI structure. Niek can be contacted at n.wijngaards AT aimtech DOT co DOT uk for Fleixble KPI and recommendation-related questions.
Copyright © 2018, NEXES Research and Innovation Action, All Rights Reserved. The NEXES Action has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 653337. The work on the NEXES Key Performance Indicators is co-authored by the Action partners and has benefited from the constructive comments by the reviewers. See the NEXES LinkedIn group 
for an overview of NEXES colleagues. All figures Copyright © NEXES unless stated otherwise and all images Copyright © www.openclipart.org unless stated otherwise.