Our paper ‘Challenges on the interaction of models and policy for pandemic control’ has been published in the journal Epidemics.
The COVID-19 pandemic has seen infectious disease modelling at the forefront of government decision-making. Models have been widely used throughout the pandemic to estimate pathogen spread and explore the potential impact of different intervention strategies. Infectious disease modellers and policymakers have worked effec-tively together, but there are many avenues for progress on this interface. In this paper, we identify and discuss seven broad challenges on the interaction of models and policy for pandemic control. We then conclude with suggestions and recommendations for the future.
Hadley L., Challenor, P., Dent, C., Isham, V., Mollison, D., Robertson, D. A., Swallow, B., Webb, C. R. (2021) ‘Challenges on the interaction of models and policy for pandemic control’, Epidemics, 37, https://doi.org/10.1016/j.epidem.2021.100499
I’ve built a model to show the concept of herd immunity. It shows why we need to not leave hard-to-reach parts of the population unvaccinated.
Herd immunity, also called population
immunity, is the protection for the population that comes from when a
proportion have been vaccinated. With more vaccinations, we move towards
this herd immunity threshold. (We’re not there yet, even though some
say we are.)
Here’s my model. Imagine a population in a country.
People are either susceptible (they may not be vaccinated or have had
the virus). We colour these green. People may have had the vaccine.
These are blue. And there’s one (near the bottom left in purple) that is
Now, in my model, they will infect anyone who is susceptible (green)
within that little circle surrounding them. And those will infect people
After a few generations, more and more are infected.
almost all (although not all – some are fortunate) are infected. That’s
bad. And it’s because we haven’t reached the herd immunity threshold.
– what happens when we reach the herd immunity threshold (or get close
to it)? We have many more vaccinated (they’re blue). Let’s see what
happens when the infectious person (this time in the middle) infects
Well, in this case, there’s a local infection, but the infection can’t be sustained (that’s good). Herd immunity.
What happens when those vaccines are not spread out equally across the country?
Let’s vaccinate *the same number of people* just in the bottom half of our population.
There we are – lots of vaccinated people in blue. That infection (right middle) doesn’t stand a chance.
let’s see what happens when that infectious person is in the top half
among the unvaccinated population (remember, we have the same number of
vaccinated people in the population as a whole).
Well, that infection spreads…
… and spreads …
… and spreads …
until huge numbers of people are infected *even though we have overall reached the herd immunity threshold*
And that is why we need to vaccinate evenly, not leaving pockets where infection can spread.
This was originally published on Twitter on 9 January 2021.
The Government is making the same mistakes as it did in the first wave. Except with knowledge.
The Government’s strategy at the beginning of the pandemic was to ‘cocoon’ the vulnerable (e.g. those in care homes). This was a ‘herd immunity’ strategy. This interview is from March.
This strategy failed. It is impossible to ‘cocoon’ the vulnerable, as Covid is passed from younger people to older, more vulnerable people. We can see this playing out through heatmaps. e.g. these heatmaps from the second wave.
The Government then decided to change its strategy to ‘preventing a second wave that overwhelms the NHS’. This was announced on 8 June in Parliament. This is not the same as ‘preventing a second wave’.
The Academy of Medical Sciences published a report on 14 July ‘Preparing for a Challenging Winter’ commissioned by the Chief Scientific Adviser that set out what needed to be done in order to prevent a catastrophe over the winter period.
Around this time, the Great Barrington Declaration was published. This changed the rhetoric from ‘herd immunity’ to ‘focused protection’. This was and remains non-mainstream from a scientific point of view, but was popular amongst a group of commentators.
Cases were increasing rapidly, and SAGE (the Scientific Advisory Group for Emergencies) called for a ‘circuit breaker’.
At the same time, a focused protection-supporting group of scientists were invited to Downing Street to present the alternate view. Politically, this enabled this headline to be written.
A new variant was discovered with increased transmissibility. This caused a ‘lockdown’ as the NHS was now in crisis mode and there was ‘a material risk of the NHS in several areas being overwhelmed’. The Government’s second strategy is failing.
However, this ‘lockdown’ is not stringent. Many more people are allowed to send their children to school allowing the virus to mix in children and their parents.
The Prime Minister made a strange comment on 6 January. The word ‘cocoon’ was back. Remember that from March?
Matt Hancock has now given an interview setting out the Government’s new, third, strategy.
The strategy is about ‘manageable risk’. Risk to those that may die or live with the effects of catching Covid, or risk to the Government?
We are now back to vaccinating the vulnerable – focused protection if you will.
It appears that the Government has adopted a hybrid strategy – vaccination for the ‘vulnerable’, and herd immunity or focused protection for those that are not. There is no discussion of vaccinating children and the under-50s. It is clear that many more lives will be lost.
Update 10 January 2021
I am pleased to say that Matt Hancock stated today that all adults will be offered a vaccine by Autumn. This is good news. There is a race between vaccination and infection. It is important that restrictions (the ‘cry freedom’ quote) are not released prematurely.
We would all like to be able to predict the future, but in order to understand the future, you have to understand the past. So is the quandary facing modellers, data analysts, and policy makers. At the heart of the decisions facing politicians are an epidemic that is doubling, delays, and the very real risks of indecision.
In the hastily brought forward press conference last
Saturday, the Prime Minister, flanked by the Chief Scientific Adviser, and the
Chief Medical Officer, solemnly stated that the country needs to go back into an albeit less stringent ‘lockdown’. How had he come to that conclusion, and what
models and data had precipitated that press conference?
Looking forward into the future, is SAGE, the Scientific
Advisory Group for Emergencies. But
beneath SAGE are several groups, one of which is SPI-M-O: Scientific Pandemic Influenza
Group on Modelling, Operational sub-group. As well as estimating the
reproduction number, R, they also project the course of the epidemic.
The Civil Contingencies Secretariat, part of the Cabinet
Office, commissions SAGE and SPI-M-O to produce an estimate of the Reasonable
Worst Case Scenario number of deaths, used for planning purposes – the idea
being that you don’t want to prepare for the worst that could happen, but
the worst that could reasonably happen. A reasonable worst case scenario
is not what you think will happen, but what you want to stop happening. Only policy can do this, not modelling.
Long Term Projections
The slides that accompanied the Number 10 press conference
last Saturday included a slide showing projections from early October charting
scenarios if there were no changes in policy or behaviour and under a number of
assumptions: R remains constant, contacts increase over winter, no additional
mitigations over and above those in early October when the projections were
made. It’s important to read the small
print on these slides, as many clues lie therein. Firstly, it shows preliminary, long term
scenarios, and secondly, each independent modelling group doesn’t just forecast
a trajectory, but a range of possible outcomes, some worse, some better. All of these modelling groups showed projections
where the daily deaths from Covid exceeded those of the first wave peak number
of daily deaths. Taken together, the
output from the models show daily deaths peaking during December at a number
greater than in the first wave. The peak
is useful for knowing the pressures on the NHS, but it’s the area under the
graph that is the most sobering – this shows how many people could sadly
One group, PHE/Cambridge, had a projection that was much
larger than the others, but importantly, SPI-M was not asked to prepare a consensus
projection for daily deaths. As
projections go further into the future, they become less certain – think of the
reliability of forecasting next month’s weather as opposed to forecasting
tomorrow’s weather. This is especially
true when we are dealing with doubling – things can (and have) got out of hand
very quickly, and small changes can have large effects.
None of this is new, of course. The Academy of Medical Sciences produced a
report in mid-July, Preparing for a Challenging Winter, which set out a
reasonable worst case scenario number of deaths (excluding those in care homes)
of around 119,000, over double the number in the first wave. But crucially, it also included priorities
for prevention and mitigation, including expanding the test, trace, and isolate
system in order that it can respond quickly and accurately; and ‘maintaining a
comprehensive, population-wide, near-real-time, granular health surveillance
system’. This of course did not happen,
with testing capacity exceeded by demand in late August, leading to
deterioration in data quality – data that in turn informs the models.
Medium Term Projections
As well as long term projections, which are useful for
long-term Government planning, medium term projections are made, firstly for
daily hospital admissions, and secondly for daily deaths. These also show a central projection, a line,
and a range for the projection. These slides were revised after the
Saturday press conference, and now show a smaller shaded area, but the central
projection, that solid line, remains the same.
These projections were more recent than the long-term
scenarios, as they are needed for immediate planning in the NHS, and therefore
need to be updated more frequently. They
are not forecasts or predictions, but represent a scenario where the epidemic
is going if it follows current trends.
They do not take into account future policy changes or behaviour
changes. It is impossible for a model to
know what will happen, but it does take into account a scenario, what
could happen. Take for example, West
Yorkshire going into Tier 3 (Very High) restrictions. This was due to come into effect last Monday,
but it didn’t. Should models have taken
that into account, even if ministers were also uncertain?
But how do we know that the underlying modelling is good? We
can look at earlier model projections and see how they coped against the actual
– not modelled – data they were projecting.
Here is an earlier version of the hospitalizations graph produced on 6
The red dots show data from before the projection was
produced. From the projection date, we
see, as with the hospitalization projection presented at the press conference,
a central projection, the line, and a ranges for the projection – in this case
two, a central 50% prediction
interval, and a wider, 90% prediction
interval. Importantly, we also see black
dots – this is the real data plotted on the graph after the projection was
made. We can see that the black dots
track the blue range very closely. (This graph is plotted on a log scale: 100
to 1,000 to 10,000 instead of 200 to 400 to 600, which some people find easier
to interpret, particularly when we are dealing with exponential growth –
doubling, or indeed exponential decay – halving.)
Models are projections into the future – they have
assumptions, and these assumptions should be made clear. There is also another set of models that
inform policy – economic models. We
haven’t seen those. And although we have
seen the minutes and the papers of SAGE and SPI-M, we haven’t seen the minutes
of where decisions are actually made, where the advice is considered and policy
made. Those decisions are made in the
Cabinet Office Briefing Rooms – COBR. Decisions
made within those walls are not so transparent: Advisers advise, and Ministers
decide. And deciding to do nothing,
particularly against scientific advice, is a decision in itself.
It wasn’t meant to be like this. Remember the Alert Levels (the ‘Nandos chart’)? The whole idea of that was to set some sort of policy – a roadmap if you will – of how we get out of a national lockdown. Introducing… Covid Alert Levels. 12 May 2020.
And remember this? The gentle ski slope of calm. ‘R less than 1 caseload decreasing’ Ah, simpler times.
This was about the same time as the ‘science bit’.
We’ve also had the shiny new: – Joint Biosecurity Centre – National Institute for Health Protection – The Contain framework and lots of other things that seem to have been good ideas at the time.
That Contain framework was a good one. Basically if you get to 50 cases per 100,000 your local authority gets on to a watchlist and restrictions come in. Leicester took the brunt of this one.
Each week, a watchlist was created of the local authorities that were nearing the 50-or-so cases per 100,000 threshold, and they were placed on a list. New restrictions came into force. Trafford was Mentioned in Dispatches with only 32 cases per 100,000
Of course, experts have been banging on about the reckless way that restrictions were lifted – with no basis in science. 10pm closure times for pubs, that sort of thing. Of course, SAGE has experts coming out of its ears, who could have advised on this sort of thing
Of course, they *did* advise on this sort of thing. Here’s the warning to the Government from ‘The Science’ on 14 July (worth a read).
Despite warnings from the Deputy Chief Medical Officer that releasing restrictions too early will backfire spectacularly, the Government pressed on relasing things. We were happy, we ate out to help out. We spread the virus.
Turns out that focus groups and popular policies aren’t necessarily the best way of doing strategy. (I wrote about this in the Financial Times in March).
So, the Government came up with a new shiny Tier Level thing. A new Nandos chart, with extra mild and mild removed. Like the Nandos chart, we start with ‘medium’. Sounds lovely. That’s most of the country.
Problem is that *every single local authority in the country’ has a higher than Trafford that managed to get itself on the Contain watchlist.
So, we now have High and Very High Tier levels. Problem is, of course, that all this procrastination means that the virus is still doubling. If the growth rate is 5%, that means doubling every two weeks.
So, we’re seeing – patients admitted to hospital increasing 30% week on week – deaths increasing 50% week on week
And of course, Test and Trace has failed completely.
We’re now back to another mess of Very High Tier 3 restrictions, not uniform at all as per the plan, but different depending on where you are.
And… we’re back to where we started – uncontrolled virus spread with a mess of differing local restrictions. Maybe we could do some Powerpoints and Blue Sky Thinking and re-invent the Nandos scale and call it the Scoville scale. Because things are getting extra hot out there.
The failure of the government’s testing strategy (Report, September 22) is a lesson in confusing resources with capabilities. Commercial NHS test and trace has resources but not capabilities. NHS labs and local authority directors of public health supported by Public Health England have capabilities but not resources.
In order to resolve this conundrum, we should provide the experts who are custodians of those capabilities with the resources they need to do their job.
Duncan Robertson School of Business and Economics, Loughborough University Leicestershire, UK
Long Interview on BBC Five Live debating against a herd immunity strategy
Interview on LBC (Nick Ferrari) and LBC (Iain Dale)
Here is the heatmap of cases for PHE week 41 using week 40 data.
Studies in Spain, France, and the US have all shown that although the second wave may start in young people, it will inevitably move to older people.
The remarkable thing about this disease is that the death rate increases massively with age.
Students are unlikely to die of Covid-19, although some may, and we are still unsure about the long-term health consequences from catching the disease.
The heatmap of cases shows how the disease has travelled through the age groups. As you go from left to right through the weeks, you can see a gradual rise upwards through the population.
These figures should be seen as a minimum. Lack of testing capacity has meant that not everyone can get a test. For example, we do not know whether delays in testing may be concentrated in certain groups such as care home residents.
The latest figures (which will be revised upwards as new cases are recorded) show a very worrying number of cases in the over-80s.
A case rate of 53 per 100,000 over-80s is very concerning. The Department of Health and Social Care have this week stopped publishing the COVID-19 surveillance report which broke down numbers of people with the disease. However, we can estimate that over 1,000 over-80s tested positive last week. Given the very high fatality rate in over-80s, we can confidently predict that over 100 over-80s will die of infections caught in the last week.
This is one of many reasons why interventions are so critical – by not clamping down hard on the disease now, we will sleep walk into a situation as bad or worse than the first wave. The mid-July Academy of Medical Sciences report commissioned by the Chief Scientific Adviser set out a reasonable worst case scenario of 119,000 deaths in this second wave excluding those in care homes. We have a choice as to whether we as a nation repeat the mistakes of the past.
We also know that we are not doing enough testing as the positivity rate is so high (7% overall for Pillar 2 tests and up to 15% in some areas such as Liverpool) (see this thread)
So, how do we go about estimating R? Here’s a post I wrote in January explaining R in relation to Covid-19 (which then didn’t have an official name) in relation to Covid-19 (which then didn’t have an official name)
To estimate R, we carry out surveys – which means you pick a representative group of people, either households or individuals, and test them repeatedly. There are two main surveys: ONS and REACT
ONS excludes student halls of residence, as ‘only private residential households, otherwise known as the target population in this bulletin, are included in the sample. People in hospitals, care homes and other institutional settings are not included’. This is confirmed here.
The REACT survey uses GP lists to generate its sample of people who are tested. But of course, new students are only just registering with their GPs, and it is unclear when the GP lists were pulled for the latest study (Round 5 of REACT-1, 18-26 Sep)
We know that halls of residence are a significant driver of transmission.
We may be systematically under-sampling from halls of residence and therefore systematically underestimating R.
The Wall Street Journal is reporting that “New York City on Wednesday will close public schools and nonessential businesses in parts of Brooklyn and Queens that have registered a week-long spike in coronavirus cases”
Let’s look at New York and then compare to a UK city, Liverpool.
Cases are high in some New York boroughs. Up to 216 cases per 100,000 per week. But school closures are also being implemented in areas with 89 cases per 100,000 (source: New York Times)
Let’s compare with Liverpool. Here is the latest @PHE_uk report. Liverpool has cases of 238 cases per 100,000 in a week. Which is slightly higher than the highest rate ZIP code in NYC.
But remember, Liverpool’s figures are for the whole local authority.
Let’s dig a little deeper into Liverpool. Here’s the map. We can see some areas with incidence in excess of 1200 cases per 100,000. That’s very high. And don’t forget this is detected cases. The number of cases will be much higher.
But how do we know that there hasn’t been enough testing? We look at positivity. Positivity is the number of people who test positive divided by the number of people tested. And this is what NYC uses to determine whether schools should be closed.
If an area of NYC has positivity greater than 3% – three in every 100 tests being positive – then schools close. What does positivity tell us? Whether enough tests are being performed.
“the World Health Organization recommended in May that the percent positive remain below 5% for at least two weeks before governments consider reopening.” (Johns Hopkins University)
So, given that positivity is set at a threshold of 3% for school closures in NYC and WHO suggest 5% before reopening, this begs the question – What is the positivity in Liverpool?
Just under 15%, according to the latest published data (PHE week 40 reporting). Which means that around 15% of all tests in Liverpool come back positive. That’s *very high*. And means not enough testing is being carried out. And this is a problem.
This is just an example of a city with large positivity. Extra testing capacity has been sent to Liverpool presumably due to students returning to universities there. This is not a Liverpool problem – it’s a national problem.