How Not to Compare Apples and Oranges

By   July 4, 2016
0

Key Performance Indicators (KPIs) are a well-known instrument to assess organisational performance. NEXES investigates the use of KPIs to assess the impact of her solutions on emergency services in different countries. The traditional usage of KPIs is to deploy a “one size fits all” strategy for an organisation. However, emergency services find this unfair, disrespectful and demotivating. What’s going on here?

This is the second article in a series of six about the NEXES Key Performance Indicators. The previous article described the proverbial ‘comparing apples and oranges’ challenge that NEXES addresses.

A renowned approach to assess organisations’ performance is to use key performance indicators (KPIs). KPIs are often employed for this purpose and even used to direct organisational change. Basically, a KPI measures ‘what you think the organisation should do’. Often KPIs are accompanied by a management team that issues commands to their employees such as: “do what you need to do and achieve these KPIs!”.

Clearly there is a danger here: stubborn efforts to achieve a KPI may result in dire problems for an organisation. Consider a company with orchards selling fruit. If a set of KPIs focuses only on high levels of sales of particular apple types and sizes, there is the risk that stubborn achievement of those KPIs can lead to a failure in entering new fruit markets. The time and energy spent on continually trying to improve the narrow set of targets may not be in proportion to the return on investment. Moreover, new endeavours, for example, growing and marketing a type of apple especially aimed at bakeries, may face disapproval at an early stage or even be ‘stopped’ for not contributing to the set of KPIs that are the main focus of the managers. Thus the orchard company may miss out on a whole new market. KPIs are powerful tools …


Another issue with KPIs became apparent when exploring the application of KPIs within NEXES. The initial idea was: can we use one set of KPIs for all emergency services? This makes sense: by defining one (albeit) large set of KPIs, it becomes possible to assess the impact of the new Internet-enabled communication technologies, compare the new performances of emergency services and ultimately give compliments to emergency services that implement these new technologies.

To draw on the apples and oranges metaphor: this is akin to defining a KPI that states that all good fruit has an orange colour. Clearly this makes it simple to compare different orchards. Obviously, this KPI holds for oranges, but does not hold for apples per se… Does this imply that apples are not good fruit?


When testing the waters with those KPIs, it became apparent that this ‘one size fits all’ approach was not deemed useful at all by emergency services (our ‘apples and oranges’). A quick round of feedback showed that those KPIs had three major flaws:

  1. Unfair. Those KPIs are assessing impact that may not be provided by an individual emergency service. Given the diversity among emergency services, there is often an emergency service that, currently or in the near future, does not include specific technologies and its impact. It is considered unfair to evaluate an emergency service on a topic that is not their own topic.
  2. Disrespectful. Those KPIs are not taking individual differences into account. Consider emergency services in rural areas: different issues play a role, such as unreachable territory and sparse communication networks that are an important factor in e.g. first responders’ arrival time at an incident location. It is considered disrespectful to ignore individual differences.
  3. Demotivating. Those KPIs are predefined and imposed on emergency services. Emergency services do not have any say in the parameterisation of those KPIs. By reducing the active participation of emergency services, there is a big chance that there will be less incentive to participate at all. Emergency services should be supported to use these new technologies in an as easy manner as possible. It is considered demotivating if emergency services cannot participate in the tailoring of those KPIs.

Clearly those KPIs were not going to be useful. As a result, the “one-size-fits-all” approach to KPIs was abandoned. A creative solution is needed, one that is fair, respectful and motivating. The next blog in this mini-series explains the novel idea.

This blog is number 2 in a series of six articles on the NEXES Key Performance Indicators and Effect Measurements. When you wish to delve deeper into the NEXES Action and its solution to comparing apples and oranges we recommend to read the deliverable D2.4. Below is the list of all the articles in the series:

Photo of Niek Wijngaards. Dr. Niek Wijngaards works for AIMTech Consulting Limited in the United Kingdom and True Information Solutions in the Netherlands as senior consultant and solution architect. His focus on user-centered innovation and his work on intelligent systems and scenario-based robust decision-making provides a sound basis for the development of the NEXES flexible KPI structure, making it possible to rigorously compare apples, oranges and indeed the entire contents of a fruit basket. Niek can be contacted at n.wijngaards AT aimtech DOT co DOT uk for KPI and fruit-related questions.


EC FlagNEXES action logo Copyright © 2016, NEXES RIA, All Rights Reserved. The NEXES Research and Innovation Action has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 653337. The work on the NEXES Key Performance Indicators is co-authored by the Action partners and has benefited from the constructive comments by the reviewers. See the NEXES LinkedIn group LinkedIn Logo for an overview of NEXES colleagues. All images Copyright © NEXES unless stated otherwise.