+44 7957421933
dave.wilson@effectivefacilities.com

The problem with Benchmarking (and some solutions ….)

“We face danger whenever information growth outpaces our understanding of how to process it”[1]

This is the dilemma we face in benchmarking Facility Management services: so much data is available, from so many diverse sources, that it is sometimes hard to know where to start. And, given that some of it is of dubious reliability or relevance, and that it is  framed and presented in different ways over different time periods it is often incomprehensible often to even us specialists, let alone to our customers and to the senior managers to whom we report and who question our contribution to the organisation.

Yet Facility Managers are encouraged to believe that all their problems will be solved if they can just present better benchmarks. This despite years of FM practice resulting in little or no effective co-operation in benchmarking, little publicly available data, no agreement on processes, costing models, quality and performance measurement … in fact, flying in the face of all logic, it seems, we still perceive Benchmarks as a panacea. They are not, of course. But if we can’t use the benchmark data we have to improve your performance what is the point of it? If it tells us what levels of performance we reached but cannot explain why that happened, we cannot guarantee either to replicate or to improve on that. So what would be the point of benchmarking? Surely we do it, if we do it at all, to allow us to analyse, understand and improve what we do?

So what we need it to understand a little better when, how and why to use the techniques of benchmarking to understand and improve our facilities performance. This article aims to help with that process of understanding.

Data shortcomings

So let us consider the shortcomings of some of the data we have to hand; then think about our problems in using that data; and then try to construct some solutions which might be applied – although I should say at this point that I don’t believe in the usefulness of a single “universal” benchmark, so any application of benchmarks that I suggest will be scenario specific. Let me explain why, by starting at the beginning: why Benchmarking is problematic for us in the FM industry.

It is always nice to look back on our performance in a glow of satisfaction. In an industry where we often feel under-appreciated and only get noticed by our bosses when things go wrong, being able to show that we did a good job – to demonstrate that to ourselves, our support teams, as well as our customers – is certainly a nice thing. But while Benchmarking can help us do that with some sense of justification, the problem is that it is wholly retrospective: it looks back at what we did. While we might suppose that is useful, actually on its own it is not especially worthwhile: we must be able to show conclusively not only that the performance will be replicated in the future, but that it will improve relative to the benchmark normative data we used for comparison. That is important because that baseline can of course change (and in both directions, too). But if these points are not considered then beyond self-satisfaction it is not clear what backward looking benchmarking alone tells us that is useful. That’s especially the case if the activities we benchmark create little or no  beneficial effect on the performance of the entire organisation: we may think we did a good job, but in effect we haven’t really achieved anything.

In addition, most benchmarks are a one-off snapshot of performance. Stripped of longer term context, in particular trend analysis, they run the risk of being misleading. That is not only because the situation they reflect may have changed but because there is no guarantee that the situation was in any way typical of normal operating conditions. As we have all experienced this year, our highly variable weather alone can have a significant impact on costs, but if we applied benchmarking without recognising that context or having an understanding of longer term climate trends then any conclusions reached would be misleading.

Another concern is that we too often assume that the data used for the benchmark – both our own performance and that of the comparator group – are accurate or relevant to the organisation. For reasons I’ll go into shortly, simply knowing that our operating cost per desk was £14,400 per annum when the current “norm” we’ve seen in the FM press is £15,000 is not particularly illuminating: we may know how we arrived at our cost figure but we have no idea how the published figure was arrived at or what’s included in it. In fact, even if we can find out what’s in the figure we use for comparison, it is extraordinarily difficult to be sure that the sums are really “like for like”, since accounting policies are so variable across organisations. That is compounded by the various ways in which facility services are packaged and the convoluted pricing and cost allocation methodologies used by building occupiers and service providers. All that leaves the reliability of data as highly questionable, and if our facility’s performance is only better by a small margin (4% in the cost per desk example above) then that may not only not be accurate, it might be wholly misleading.

In a similar way, another problem is that we often benchmark data that is easy to capture rather than what is relevant. Nate Silver, again:

“The instinctual shortcut that we take when we have ‘too much information’ is to engage selectively, picking at the parts we like and ignoring the remainder…”

So we can often see published benchmarks expressed as per square metre costs, as if this was all that mattered. Where, to illustrate the complexity we avoid, are the per cubic metre benchmark costs for heating, for example? We don’t find those published because while everyone (often wrongly) thinks they know the floorplate of their properties, no-one knows the internal cubic space dimensions, yet fairly obviously heating costs are driven by cubic space not floor plate. Equally we don’t find many sources for comparable data which reflect sustainability performance, or impacts on staff productivity or staff turnover, or utilities efficiency ….

Considering how critical to most of the outputs of most organisations the delivery of effective facilities operations is, and how complex the relationship between any two elements is, perhaps it is clearer how a misalignment might occur. Even in the most simplistic interpretation of commercial value it is difficult to show what it is that FM operations add to the “bottom line” of the organisation, and difficult to separate out our contribution from that of all the other supporting functions. But “bottom line” is what senior management think about all the time.  So logic suggests that, as with the cost per desk example above, we should focus on a few key indicators that demonstrate our efficiency. The problem there is that efficient space utilisation is not the same as providing an effective productive workplace: in fact many experts think that the obsession in the UK with operating cost per square metre has diverted us from the true value of what we do and led to the commoditisation of our profession. It can also prevent us from recognising the importance of doing things which are not easily measurable: as someone notably said on Twitter only this week: “what is the Return on Investment of putting your trousers on?” All that compliance stuff we have to do carries a cost, but the return is in risk avoidance or mitigation  which even the most astute and experienced managers find difficult to measure.

The next drawback is the sheer impossibility of finding valid comparators for most buildings: the European benchmarking standard[2] contains almost two pages worth of variable factors to be considered in International benchmarking (shown in Table One), most of which also apply within the UK.

Table One: Inherent Complication in Benchmarking

Screen Shot 2013-06-24 at 19.58.01

To show the complexity, let me quote just one of those in detail:

use of occupancy (take into account: type of activity undertaken in the buildings; effect of potential alternative use; costs of adaptation for different use; any impact/inefficiency in core business resulting from the building design; provision of any non-standard services [for example health club / gymnasium / sports facilities; social facilities; restaurants; data centres or mission critical activities; specialist storage facilities; very high levels of security provision])

All this points towards another related problem: even if we try to follow the solution offered by the European Standard EN15221-7, which is to identify and analyse just one factor and seek suitable comparators against which to measure our own facility’s performance, the sheer variety of possible options is confusing. There are literally hundreds of possible measures we might apply, and yet there is no guarantee that we can find appropriately like-for-like data to match against them: simply put, the level of detail (granularity, if you prefer) and the resources we have access to internally is not matched by easily accessible comparable data from peers.

Which brings us to the final data related problem: that even when you’ve done all this and gathered some basic data that you think is reliable, accurate, relevant and so on: how much time and effort has it taken? How do you balance that cost (real cash or opportunity costs of your time) against the improvements that maybe, just maybe, might flow from whatever insights you glean from that data? In other words: there is a serious possibility that the cost/benefit relationship between benchmarking and resulting improvements doesn’t justify the effort. Which, perhaps, your line managers might come to think you should have considered before spending your time trying to benchmark the services.

So let me summarise the problem we face: The way we approach benchmarking is an attempt to reduce the complexities of FM operations to a single number: one indicator that tells us, and the world, how well we do our jobs. But FM isn’t so simple that we can achieve this in any meaningful way. The work we do and the facilities we manage are too complex to be explained or measured with any consistency by one indicator.

Problems in using data

Not that data gathering is the only problem. Perhaps more vexing for those of an analytical mindset is that quite often data is mis-represented or misunderstood.

Let’s assume we have managed to obtain some reliable, accurate data about peer facility performance. Now we want to carry out a comparison, of course.  But what precisely is our qualification for analysing data? Have we undertaken a course in statistics? Do we really know how to make use of the data? The chances are that Facility Managers don’t, because statistics is a highly complex field, and it takes more than just being able to use a spreadsheet to analyse complex data. For example, is a single source comparator valid? If it is an average, do we know the numbers in the sample, where they came from, when they were reported, what the standard deviation is, where the upper and lower quartiles fall? If it is a range of data, do we know how to analyse that set of data to create meaningful comparisons?

Among many other problems (and all those issues I raised about data sourcing previously still remain to make us wary of the usefulness of any data questionable, by the way), key is the issue of what drives the performance that we see. That is partly a question of making sure that we have analysed the “root cause” data, rather than just one of the symptoms, or at least that we understand the underlying chain of causality. Because if we don’t understand the events which create or influence the pattern of data then there is little or nothing that we can with any of the facts we obtain because we won’t know what to change to improve. Nor will we be sure that the comparative data actually apply to our situation. If we don’t have that confidence then we cannot seriously begin to use the information as a basis for changing our operational performance. In short, benchmarking activities can be devoid of a sense of causality: simply knowing what the cost per square metre is won’t be particularly helpful if we don’t know why it is. Nor will it help if the factors that drive those costs are outside our control.

What this amounts to is the risk of blindly assuming that future performance can be predicted based on past outcomes. That’s only ever going to be roughly possible, especially given the complexity of services that come under the “FM” category. It is not a question of simply budgeting – after all, within a large budget we always have  some room for manoeuvre.

So, if we attempt to use benchmarks in a more sophisticated way than simply to compare costs, then the two key problems we face are understanding what the data tells us, and understanding how to apply that information to create future improvements. Without that ability we have simply created a number in a vacuum.

Which relates to the final shortcoming : how well do the benchmarks we create link to the desired organisational outcomes? This matters because in assessing operational performance doing something for the sake of doing it is not perceived as positive, no matter how much better than peers that performance might be. In some cases – health and safety compliance, for example – then benchmarks might be irrelevant because the target performance is already set at 100% by organisational policy and service user expectations.

Possible solutions

Having said all that, the strong desire for benchmarks in the industry suggests that there is a perceived benefit in having them. So we need to consider how to create meaningful, useful measures, how to communicate the results, and how to apply them for positive outcomes. In this, the European Standard provides a sound methodology without providing any ready-made solutions, as Table Two shows.

A brief summary of that process might be: be clear about what we want to know, work out how it can be measured, work out where to obtain the data, capture and analyse the data, and then make any improvements which have been identified.

It sounds simple when put like that. But as has been discussed, there are problems at each stage. Which is why I believe that the key to successful benchmarking is simplicity. The basis for that, I suggest, is to select an area for analysis:

  1. which needs improvement
  2. which is within the control of the FM team
  3. which will have an impact on the organisation
  4. where data is likely to be readily available

To make life simpler, there are several ways in which it can be made easier to find relevant and accurate data. Most overlooked of these ought to be the most obvious: comparison against our own performance. That could be against previous time periods  (not necessarily years – it could be months or just weeks in some cases) or by comparing performance between locations. Self-comparison provides some of the best data, with the most ease and greatest speed. Not only should the information be reliable, complete and accurate,  but because the constraints of the operations and the sites will be known it is easier to interpret the data. Finding outliers (best and worst performers) and understanding what caused that will also be easier, since internally there is less risk of the fear of exposure of poor performance being a barrier to data access. At the same time, be wary of trying to move all performance to the “best in class” level: not only may that not be achievable, but it may be disruptive and excessively costly to get to that level, when what is required is steady improvement and travel in the direction of the best. Put another way: we cannot all be in the top quartile of performance.

Table Two: BS EN15221-7 Benchmarking Process

Screen Shot 2013-06-24 at 20.00.23

Adopting this approach offers a more ready application of continuous improvement as well, since because the data is easier to obtain it is also easier to “read” the results more often, so the impact of changes can be seen in shorter timescales, which the frequency of external data collection renders almost impossible.

However, there is one opportunity which is likely to be missed doing this, and it is one which I believe a lot of benchmarking processes also omit. There is an obsession with comparing “like with like”, which I have referred to and to some extent advocated in this paper. But, provided we know what we are looking to achieve,  it can also be highly informative to compare with operations that are not like our own. That can apply to comparing processes as well as other performance features. For example, how do waiting times in hotel reception areas compare with those in office receptions? If we can legitimately compere our service quality aspirations with hotels, is there something there to learn about visitor handling which might improve our own service? Our obsession with looking for like-for-like or peer comparators  creates the risk of preventing us for looking to see whether what we offer is genuinely best practice, and whether we can learn from norms in other spheres of activity. Like-for-unlike can be informative in the right circumstances.

But for those who are intent on measuring against direct comparators, and doing so regularly, then why not set up (or join) a benchmarking club. These already operate in some sectors (pharmaceuticals and petro-chemical industries for example), and there are also some area and regional clubs operating through local Chambers of Commerce and similar organisations. The caveats I issued against data comparability, especially at the more granular level do apply here though: data that is shared has to be assumed to be accurate, since you cannot test it, and often it is agglomerated and averaged before being made available, which makes analysis more difficult. But it can provide some insights and offer some suggestions about areas to improve. And, happily, there are a number of possible commercial sources for data comparison which you can access. Those include IFMA’s benchmarking service (http://www.ifma.org/know-base/research-benchmarks-surveys);  BIFM’s  partnership with FMBenchmarking (https://www.fmbenchmarking.com/whitepapers) ;  Leesman Index (http://leesmanindex.com); and IPD  (http://www1.ipd.com/).  Each of these has different characteristics and strengths, but all offer excellent access to well validated and up-to-date information sources.

Finally, let’s not forget that good data is gold-dust. When it is created it may not be when it is needed, but reliable historical information is essential to understanding not only trends but also to ensuring that any “snapshot” of performance is not itself an exception to normal performance levels. So “banking” data – capturing and maintaining information which can be captured now, from a range of sources, for future use is both fundamental to having good sources of data, but is an important safeguard against mistaking short-term variations for a longer term trend in performance.

In summary, benchmarking is often portrayed as a quick and simple fix leading to service cost and quality improvements . It is clearly nothing of the sort. But with a genuine understanding of what we are trying to achieve, application of a careful methodology and the development of solutions based on the resulting facts, it can not only improve services but be a great tool for communicating outcomes to customers and, more importantly perhaps, senior management. When benchmarking becomes systematic and embedded in FM processes, it of course becomes easier – and it can then be the foundation of a genuine continuous improvement regime. It is something which the industry should be doing much more of, for the benefit of us all.

© Dave Wilson, 2013



[1] The Signal and The Noise, Nate Silver (Penguin Books, 2013)

[2] BS CEN 15221-7,  available from BSI at http://shop.bsigroup.com