Playing the ‘what-if’ game

By   April 23, 2018
0

Monitoring & Planning with Flexible KPIs

Given the multi-stakeholder context of emergency services, the evaluation of NGES performance must accommodate multiple stakeholders and their possibly different priorities on (part of) NGES performance. The evaluation framework provides a new perspective that reflects the technological possibilities that have become real or already materialised and that can be put into use to increase the safety and security of citizens, make the respective needs and the services provided better meet each other and, thus, increase the professionalism, efficiency and effectiveness of emergency response organisations.

The adoption of NGES must use an evaluation framework that accommodates multi-stakeholder perspectives and priorities, even allowing re-evaluation from the perspective of different stakeholders. Emergency Services must use this evaluation framework to support determining tactics in technology uptake, development and other activities relevant to NGES adoption.

The Flexible KPIs are a means for enlarging mind-sets and extending horizons of thinking about and measuring NGES performance. The Flexible KPIs can assist an emergency service organisation on its path of continuous development so that, eventually, the full potential is gained from the new technology for the benefit of all.

The adoption of NGES should make use of the Flexible KPIs that support monitoring, ‘what-if’ explorations, retrospective re-evaluation, (re)planning activities of an Emergency Service organisation performance as well as comparison of multiple Emergency Services’ performance. The flexibility of the KPIs ensures structured comparison of customised evaluation of NGES, non-fully NGES and non-NGES implementations and should be used to support continuous development and improvement of Emergency Services.

Providing recommendations for the adoption of NGES technologies requires offering support to emergency services in technological and organization change processes, given commonalities and respecting differences among emergency services. From the outset, we can presume that each emergency service organisation is likely to adopt their own individual roadmap.

The example organisations introduced in an earlier article each have their own technology replacement roadmap. Organisation A (rural) is currently experimenting with its own emergency app, and has reserved budget and other resources for replacement of outdated technology: it can plan for relatively ‘early’ adoption of NGES technologies. Meanwhile, organisation B (urban) has recently invested in updates of its technology and considers a more complicated adoption plan involving first a pilot in an industrial district before having confidence in and persuading policy-makers for a city-wide adoption of NGES technologies.

The NEXES Flexible KPIs are aligned with the commonalities among emergency services and have been engineered to be adaptable to the differences, accommodating the individuality of each specific emergency service organisation. These Flexible KPIs allow measurement not only of quantifiable aspects (e.g., number of video calls) of NGES impact, but also qualitative aspects (e.g., improvement of quality of information sharing, opinion of citizens on incident mitigation).

Planning and re-planning

The Flexible KPIs are well-positioned for planning and re-planning activities. Although in an ideal world a plan can always be executed to perfection, everyday practice teaches us that plans often need to change. The Flexible KPIs can support changes to plans:

  • Monitor progress over time: by collecting measurement data, continuously or at specific moments in time, the measured data can be evaluated.
  • Over time, effect measurements may change in relevance and importance. The Flexible KPIs are designed to allow structured comparison within the same emergency service organisation over time, or even between multiple emergency services.
  • The Flexible KPIs are technology agnostic and assist in providing an objective socio-technical evaluation of progress, irrespective of technologies used and/or (in)actions taken.
  • Re-evaluation of Flexible KPIs is built-in from the start, this enables inspection of the past with contemporary priorities, as well as being able to engage in ‘what-if’ exploration of the past, leading to more-informed decision making about the future.

The NEXES Action does not recommend a specific planning methodology. Rather, the Flexible KPIs are offered as an instrument to be used in the planning methodologies of emergency services. Almost every planning methodology can be considered as a type of OODA loop: by showing how the Flexible KPIs relate to the OODA loop, the usage of the Flexible KPIs becomes apparent in monitoring and planning:

OBSERVE
The Flexible KPIs require data to be collected. The Effect Measurements assist in making explicit: what data should be collected, from what sources, including systems, emergency services’ professionals and citizens.
ORIENT
The Flexible KPIs provide the mechanisms to analyse and evaluate the collected data in the prioritized KPI structure. In addition, (re-)evaluation of (previously collected) data is also supported.
DECIDE
The outcomes of the flexible KPI analysis and the ability to easily investigate the effect of changes in priorities to impact, supports decision making on what actions to take.
ACT
This is not covered by the Flexible KPIs, but rather by the emergency services themselves. By restarting the OODA loop the results of acting can be observed, and taken into account in the subsequent steps.

Scenario planning

The ability to monitor a situation and being able re-evaluate the current and past situations makes it possible to engage in ‘what-if’ thinking. In a nutshell, this entails considering different circumstances in planning or decision-making situations, e.g. see scenario planning and robust decision making techniques.

For example, if policy-makers have a different prioritization of the importance of KPIs than their related emergency services, this may have a large influence on the relationship between these parties. Recognizing the need for different priorities and allowing to evaluate current and past evaluations with different KPI prioritization, makes it possible for emergency services to take into account how past and expected evaluations will be evaluated by their policy-makers, and vice versa. Being able to understand each other’s perspectives is an important factor in maintaining good relationships.

The key point is that if the Flexible KPIs are to be used as an instrument in planning activities, then the Flexible KPIs must also support ‘what if’ usage. It is of paramount importance that emergency services can consider possible plausible developments concerning their own organisation, their area of responsibility, political, technological and social developments as well as other factors when deciding on plans for technological and/or organisational change.

The Flexible KPIs have been designed to be customisable by emergency services while assuring transparency in the methods and parameters for evaluation of effect measurements as well as the prioritisations used within KPIs and when aggregating KPI scores. This makes it possible to compare KPI scores for different points in time for a single emergency service organisation, even when different effect measurements were considered (ir)relevant. Furthermore, multiple emergency services’ evaluations can be compared to gain an overall impression of NGES performance.

This article has discussed the importance of NGES performance evaluation in monitoring and planning methods of emergency services. The next article discusses whether the Flexible KPIs are proprietary property or can be freely used.

This blog is number 3 in a series of five articles on the NEXES KPIs Lessons Learned and Recommendations. When you wish to delve deeper into the NEXES Action and its solution to comparing apples and oranges we recommend to read the deliverable D2.4, for more information on the NEXES Recommendations we recommend to read the deliverable D2.5. Below is the list of all the articles in the series:

Photo of Niek Wijngaards. Dr. Niek Wijngaards works for AIMTech Consulting Limited in the United Kingdom and True Information Solutions in the Netherlands as senior consultant and solution architect. His focus on user-centered innovation and his work on intelligent systems and scenario-based robust decision-making provides a sound basis for the development of the NEXES Flexible KPI structure. Niek can be contacted at n.wijngaards AT aimtech DOT co DOT uk for Fleixble KPI and recommendation-related questions.


EC FlagNEXES action logo Copyright © 2018, NEXES Research and Innovation Action, All Rights Reserved. The NEXES Action has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 653337. The work on the NEXES Key Performance Indicators is co-authored by the Action partners and has benefited from the constructive comments by the reviewers. See the NEXES LinkedIn group LinkedIn Logo for an overview of NEXES colleagues. All figures Copyright © NEXES unless stated otherwise and all images Copyright © www.openclipart.org unless stated otherwise.