When metrics go wrong: Offering everyone a booster by January

The UK Government’s aim to have a big push on booster vaccinations is really great. But their target to “offer a booster to everyone eligible by the end of January” is a terrible target. It misses out the vital importance of takeup, is an obvious setup for fiddling the target, and fundamentally fails to address what we actually care about. With luck it won’t matter, because of the clear sense of mission.

The Omicron variant of Covid-19 is looming over the world. (Huge thanks here to the amazing efforts made by South Africa to track and identify it, and the very noble way they gave the world an early alert – they deserve a much better response than they’ve had to date.) In the UK we’re aiming to get booster vaccinations out to people as soon as we can. Last week, the Prime Minister announced a major initiative to get this done. “It’s time for another Great British Vaccination Effort,” as he put it in his inimitable style.

This is really great and I very strongly support this drive. I’ve had my booster already, and you should go get yours as soon as you can. I really really want this effort to succeed!

However, as someone who deals with metrics and targets professionally, I’m dismayed at the headline target. To quote Mr Johnson again: “The target we have set out ourselves is to offer a booster to everyone eligible by the end of January.”

This is a stupid target.

It doesn’t measure what we actually care about. We actually care about doses administered, not doses offered. Jabs that are offered but not taken up will have no effect on the virus whatsoever.

I know why people put forward this sort of target, even without any bad intention. The thinking goes: “A target you are set should be within your control. Whether people take up the offer of a jab or not is not under our control. All we can do is offer it to them. So we will set the target for the offer, not the takeup.”

There’s at least three overlapping problems with setting this sort of target.

1. Influence matters as well as control

First, it’s missing the absolutely vital point that they can certainly influence whether people take up the offer of a jab or not. How convenient and easy it is to get the jab will have a very material impact, for starters.

2. Misuse and misdirection

Second, it’s ripe for misuse and misdirection.

It’s a bait-and-switch target. People not paying close attention will hear it as “everyone will be jabbed by January”, which sounds much better than “by the 1 February the central online booking system will have been opened to allow people who can prove to the system’s satisfaction that they meet the criteria to make an appointment for a jab, but the appointments may well be much later”. That would be much worse as an outcome but would still technically meet the target. There’s also much wriggle room in “everyone eligible” since the eligibility criteria can be changed. This isn’t a theoretical confusion that could only affect the naive and uninformed: this morning (Tue 7 Dec 2021), on Radio 4’s flagship Today programme, Dominic Raab, the Deputy Prime Minister, appeared to mis-state the target as getting everyone jabbed by the end of January.

At a level down, it’s setting things up for metrics-driven stupidity. If it starts to look like the target is in jeopardy, the people accountable for delivering it will have a big incentive to hit the nominal target by detatching their efforts from hitting the thing we actually care about. Or, worse, try to hit the nominal target while harming the actual thing we care about. So, for instance, they could offer extremely inconvenient, short-notice take-it-or-leave-it appointments to everyone. That would hit the target, but would be much worse for our efforts at fighting the virus than getting more jabs in to people. Jabs offered but not taken up are even more useless than doses of vaccine left on the shelf and not put in to arms. (Those could be used later.)

And there is a sorry history of targets being missed and abused earlier on in the pandemic. It would be bad to have that again.

3. Setting up to fail

Third, this focus on only including things wholly under your control sets the whole organisation (here, the whole country) up to fail because it’s nobody’s job to actually make things succeed. If everyone has targets like this, everyone can hit their targets but the whole operation fails utterly. This isn’t a purely theoretical concern – it does happen.

Doing better

What would be better?

Spotting flaws in a target is much easier than coming up with a good one. And you can’t set a good target without understanding more of the practicalities than I have access to in this context. I am also a huge believer in developing targets with the people who will deliver them so they have a sense of ownership over them. If I, as your boss, impose a stretch target you don’t think you can hit, you may be inclined to prove yourself right. But if you agree and buy in to the target, hitting it will prove you right.

From the outside, I would prefer a target more like “A booster programme that’s at least as successful by January as the initial vaccine roll-out was”. That’s clear in broad brush, and can be cashed out at lower levels as well. So, for instance, if take-up of second jabs in a group was 80%, we would count ourselves as successful if we can get 80% of them boosted. Which is only 64%, but I think doing much better than that by January is unrealistic.

When you optimise for a metric that doesn’t include everything you care about, all the things you care about that aren’t included in the metric will be optimised out.

When you set a target, you are incentivising behaviour towards that target. You need to be very, very careful that the target is what you actually want to have achieved, or that you have some wider mechanisms or context to help avoid these pathologies.

As I like to put it in dense terms: when you optimise for a metric that doesn’t include everything you care about, all the things you care about that aren’t included in the metric will be optimised out.

Happily, most of the people involved in doing the booster programme care less about the target than they care about making an actual difference, so I’m optimistic the effort won’t necessarily be sunk by the terrible target.

The initial roll-out had a very similar target and was a huge success. The key there was the combination of an overwhelming sense of mission and purpose shared by a huge number of talented people, and the resources being made available. I remain in utter awe of what was achieved by such a wide range of folk, from scientists to policy advisers to senior health executives to pretty much all of the NHS in some way to a vast army of frontline staff and volunteers. And not just the ones on the formal registers. It was also people like Jane the veteran GP receptionist who passed up her retirement cruise to coordinate setting up the vaccination centre in the village hall, who roped in Dave’s Man And Van to fetch extra chairs from Sarah’s Wedding Celebrations, and the landlord who said “Ok fine, go for it” when people started using the pub car park as overspill.