In Defence of Models. And Modellers

This article originally appeared in The Telegraph

We would all like to be able to predict the future, but in order to understand the future, you have to understand the past.  So is the quandary facing modellers, data analysts, and policy makers.  At the heart of the decisions facing politicians are an epidemic that is doubling, delays, and the very real risks of indecision.

In the hastily brought forward press conference last Saturday, the Prime Minister, flanked by the Chief Scientific Adviser, and the Chief Medical Officer, solemnly stated that the country needs to go back into an albeit less stringent ‘lockdown’.  How had he come to that conclusion, and what models and data had precipitated that press conference?

Looking forward into the future, is SAGE, the Scientific Advisory Group for Emergencies.  But beneath SAGE are several groups, one of which is SPI-M-O: Scientific Pandemic Influenza Group on Modelling, Operational sub-group. As well as estimating the reproduction number, R, they also project the course of the epidemic.

The Civil Contingencies Secretariat, part of the Cabinet Office, commissions SAGE and SPI-M-O to produce an estimate of the Reasonable Worst Case Scenario number of deaths, used for planning purposes – the idea being that you don’t want to prepare for the worst that could happen, but the worst that could reasonably happen. A reasonable worst case scenario is not what you think will happen, but what you want to stop happening.  Only policy can do this, not modelling.

Long Term Projections

Source: Number 10 Press Conference

The slides that accompanied the Number 10 press conference last Saturday included a slide showing projections from early October charting scenarios if there were no changes in policy or behaviour and under a number of assumptions: R remains constant, contacts increase over winter, no additional mitigations over and above those in early October when the projections were made.  It’s important to read the small print on these slides, as many clues lie therein.  Firstly, it shows preliminary, long term scenarios, and secondly, each independent modelling group doesn’t just forecast a trajectory, but a range of possible outcomes, some worse, some better.  All of these modelling groups showed projections where the daily deaths from Covid exceeded those of the first wave peak number of daily deaths.  Taken together, the output from the models show daily deaths peaking during December at a number greater than in the first wave.  The peak is useful for knowing the pressures on the NHS, but it’s the area under the graph that is the most sobering – this shows how many people could sadly die.

One group, PHE/Cambridge, had a projection that was much larger than the others, but importantly, SPI-M was not asked to prepare a consensus projection for daily deaths.  As projections go further into the future, they become less certain – think of the reliability of forecasting next month’s weather as opposed to forecasting tomorrow’s weather.  This is especially true when we are dealing with doubling – things can (and have) got out of hand very quickly, and small changes can have large effects.

None of this is new, of course.  The Academy of Medical Sciences produced a report in mid-July, Preparing for a Challenging Winter, which set out a reasonable worst case scenario number of deaths (excluding those in care homes) of around 119,000, over double the number in the first wave.  But crucially, it also included priorities for prevention and mitigation, including expanding the test, trace, and isolate system in order that it can respond quickly and accurately; and ‘maintaining a comprehensive, population-wide, near-real-time, granular health surveillance system’.  This of course did not happen, with testing capacity exceeded by demand in late August, leading to deterioration in data quality – data that in turn informs the models.

Medium Term Projections

Source: Number 10 Press Conference

As well as long term projections, which are useful for long-term Government planning, medium term projections are made, firstly for daily hospital admissions, and secondly for daily deaths.  These also show a central projection, a line, and a range for the projection. These slides were revised after the Saturday press conference, and now show a smaller shaded area, but the central projection, that solid line, remains the same.

These projections were more recent than the long-term scenarios, as they are needed for immediate planning in the NHS, and therefore need to be updated more frequently.  They are not forecasts or predictions, but represent a scenario where the epidemic is going if it follows current trends.  They do not take into account future policy changes or behaviour changes.  It is impossible for a model to know what will happen, but it does take into account a scenario, what could happen.  Take for example, West Yorkshire going into Tier 3 (Very High) restrictions.  This was due to come into effect last Monday, but it didn’t.  Should models have taken that into account, even if ministers were also uncertain?

But how do we know that the underlying modelling is good? We can look at earlier model projections and see how they coped against the actual – not modelled – data they were projecting.  Here is an earlier version of the hospitalizations graph produced on 6 October.

Source: SAGE / SPI-M-O

The red dots show data from before the projection was produced.  From the projection date, we see, as with the hospitalization projection presented at the press conference, a central projection, the line, and a ranges for the projection – in this case two, a central 50% prediction interval, and a wider, 90% prediction interval.  Importantly, we also see black dots – this is the real data plotted on the graph after the projection was made.  We can see that the black dots track the blue range very closely. (This graph is plotted on a log scale: 100 to 1,000 to 10,000 instead of 200 to 400 to 600, which some people find easier to interpret, particularly when we are dealing with exponential growth – doubling, or indeed exponential decay – halving.)

Models are projections into the future – they have assumptions, and these assumptions should be made clear.  There is also another set of models that inform policy – economic models.  We haven’t seen those.  And although we have seen the minutes and the papers of SAGE and SPI-M, we haven’t seen the minutes of where decisions are actually made, where the advice is considered and policy made.  Those decisions are made in the Cabinet Office Briefing Rooms – COBR.  Decisions made within those walls are not so transparent: Advisers advise, and Ministers decide.  And deciding to do nothing, particularly against scientific advice, is a decision in itself.

A review of where we are with UK Covid restrictions. With apologies to Nandos.

It wasn’t meant to be like this. Remember the Alert Levels (the ‘Nandos chart’)? The whole idea of that was to set some sort of policy – a roadmap if you will – of how we get out of a national lockdown. Introducing… Covid Alert Levels. 12 May 2020.

Image

And remember this? The gentle ski slope of calm. ‘R less than 1 caseload decreasing’ Ah, simpler times.

Image

This was about the same time as the ‘science bit’.

We’ve also had the shiny new:
– Joint Biosecurity Centre
– National Institute for Health Protection
– The Contain framework
and lots of other things that seem to have been good ideas at the time.

That Contain framework was a good one. Basically if you get to 50 cases per 100,000 your local authority gets on to a watchlist and restrictions come in. Leicester took the brunt of this one.

Each week, a watchlist was created of the local authorities that were nearing the 50-or-so cases per 100,000 threshold, and they were placed on a list. New restrictions came into force. Trafford was Mentioned in Dispatches with only 32 cases per 100,000

Of course, experts have been banging on about the reckless way that restrictions were lifted – with no basis in science. 10pm closure times for pubs, that sort of thing. Of course, SAGE has experts coming out of its ears, who could have advised on this sort of thing

Of course, they *did* advise on this sort of thing. Here’s the warning to the Government from ‘The Science’ on 14 July (worth a read).

Despite warnings from the Deputy Chief Medical Officer that releasing restrictions too early will backfire spectacularly, the Government pressed on relasing things. We were happy, we ate out to help out. We spread the virus.

Turns out that focus groups and popular policies aren’t necessarily the best way of doing strategy. (I wrote about this in the Financial Times in March).

So, the Government came up with a new shiny Tier Level thing. A new Nandos chart, with extra mild and mild removed. Like the Nandos chart, we start with ‘medium’. Sounds lovely. That’s most of the country.

Image


Problem is that *every single local authority in the country’ has a higher than Trafford that managed to get itself on the Contain watchlist.

So, we now have High and Very High Tier levels. Problem is, of course, that all this procrastination means that the virus is still doubling. If the growth rate is 5%, that means doubling every two weeks.

Image
Image

So, we’re seeing
– patients admitted to hospital increasing 30% week on week
– deaths increasing 50% week on week

Image

And of course, Test and Trace has failed completely.

We’re now back to another mess of Very High Tier 3 restrictions, not uniform at all as per the plan, but different depending on where you are.

And… we’re back to where we started – uncontrolled virus spread with a mess of differing local restrictions. Maybe we could do some Powerpoints and Blue Sky Thinking and re-invent the Nandos scale and call it the Scoville scale. Because things are getting extra hot out there.

Image

Media Roundup week ending 12 October 2020

Letter in the Financial Times

The failure of the government’s testing strategy (Report, September 22) is a lesson in confusing resources with capabilities. Commercial NHS test and trace has resources but not capabilities. NHS labs and local authority directors of public health supported by Public Health England have capabilities but not resources.

In order to resolve this conundrum, we should provide the experts who are custodians of those capabilities with the resources they need to do their job.

Duncan Robertson School of Business and Economics, Loughborough University Leicestershire, UK

Long Interview on BBC Five Live debating against a herd immunity strategy

Interview on LBC (Nick Ferrari) and LBC (Iain Dale)

Interview on Sky News

Background to The Daily Mail, The Guardian

Heatmap of Cases & Deaths in the over-80s

Here is the heatmap of cases for PHE week 41 using week 40 data.

Image

Studies in Spain, France, and the US have all shown that although the second wave may start in young people, it will inevitably move to older people.

The remarkable thing about this disease is that the death rate increases massively with age.

Students are unlikely to die of Covid-19, although some may, and we are still unsure about the long-term health consequences from catching the disease.

The heatmap of cases shows how the disease has travelled through the age groups.  As you go from left to right through the weeks, you can see a gradual rise upwards through the population.

These figures should be seen as a minimum.  Lack of testing capacity has meant that not everyone can get a test.  For example, we do not know whether delays in testing may be concentrated in certain groups such as care home residents.

The latest figures (which will be revised upwards as new cases are recorded) show a very worrying number of cases in the over-80s.

A case rate of 53 per 100,000 over-80s is very concerning.  The Department of Health and Social Care have this week stopped publishing the COVID-19 surveillance report which broke down numbers of people with the disease.  However, we can estimate that over 1,000 over-80s tested positive last week.   Given the very high fatality rate in over-80s, we can confidently predict that over 100 over-80s will die of infections caught in the last week.

This is one of many reasons why interventions are so critical – by not clamping down hard on the disease now, we will sleep walk into a situation as bad or worse than the first wave.   The mid-July Academy of Medical Sciences report commissioned by the Chief Scientific Adviser set out a reasonable worst case scenario of 119,000 deaths in this second wave excluding those in care homes.  We have a choice as to whether we as a nation repeat the mistakes of the past.

We May Be Systematically Underestimating R by Excluding Students in Halls of Residence

We know that data on the Government Coronavirus dashboard is unreliable (see this Twitter thread).

Image

We also know that we are not doing enough testing as the positivity rate is so high (7% overall for Pillar 2 tests and up to 15% in some areas such as Liverpool) (see this thread)

So, how do we go about estimating R? Here’s a post I wrote in January explaining R in relation to Covid-19 (which then didn’t have an official name) in relation to Covid-19 (which then didn’t have an official name)

To estimate R, we carry out surveys – which means you pick a representative group of people, either households or individuals, and test them repeatedly. There are two main surveys: ONS and REACT

ONS excludes student halls of residence, as ‘only private residential households, otherwise known as the target population in this bulletin, are included in the sample. People in hospitals, care homes and other institutional settings are not included’. This is confirmed here.

Image

The REACT survey uses GP lists to generate its sample of people who are tested. But of course, new students are only just registering with their GPs, and it is unclear when the GP lists were pulled for the latest study (Round 5 of REACT-1, 18-26 Sep)

Image

We know that halls of residence are a significant driver of transmission. We may be systematically under-sampling from halls of residence and therefore systematically underestimating R.

We Are Still Not Doing Enough Testing: A Case Study of New York and Liverpool Schools

The Wall Street Journal is reporting that “New York City on Wednesday will close public schools and nonessential businesses in parts of Brooklyn and Queens that have registered a week-long spike in coronavirus cases”

Let’s look at New York and then compare to a UK city, Liverpool.

Cases are high in some New York boroughs. Up to 216 cases per 100,000 per week. But school closures are also being implemented in areas with 89 cases per 100,000 (source: New York Times)

Image
Source: New York Times
Image
Source: New York Times

Let’s compare with Liverpool. Here is the latest @PHE_uk report. Liverpool has cases of 238 cases per 100,000 in a week. Which is slightly higher than the highest rate ZIP code in NYC.

But remember, Liverpool’s figures are for the whole local authority.

Image
Source: PHE Week 40 Report


Let’s dig a little deeper into Liverpool. Here’s the map. We can see some areas with incidence in excess of 1200 cases per 100,000. That’s very high. And don’t forget this is detected cases. The number of cases will be much higher.

Image
Source: PHE Week 40

But how do we know that there hasn’t been enough testing? We look at positivity. Positivity is the number of people who test positive divided by the number of people tested. And this is what NYC uses to determine whether schools should be closed.

If an area of NYC has positivity greater than 3% – three in every 100 tests being positive – then schools close. What does positivity tell us? Whether enough tests are being performed.

“the World Health Organization recommended in May that the percent positive remain below 5% for at least two weeks before governments consider reopening.” (Johns Hopkins University)

So, given that positivity is set at a threshold of 3% for school closures in NYC and WHO suggest 5% before reopening, this begs the question – What is the positivity in Liverpool?

Just under 15%, according to the latest published data (PHE week 40 reporting). Which means that around 15% of all tests in Liverpool come back positive. That’s *very high*. And means not enough testing is being carried out. And this is a problem.

Image
Source: PHE Week 40

This is just an example of a city with large positivity. Extra testing capacity has been sent to Liverpool presumably due to students returning to universities there. This is not a Liverpool problem – it’s a national problem.

Latest Cases Heatmap Analysis: 22 cases per 100,000 over-80-year-olds

Today’s analysis shows 22 cases per 100,000 in the over-80s population in England.

To put this into context, the Government uses a rate of 20 cases per 100,000 in order to determine (with other factors) whether you should self isolate when you return from a country with this incidence of Covid-19.

An interpretation would be: if you went to a country consisting of only over-80s, you would have to self-isolate when returning to the UK.

Click here for:

Background

Why It Matters

High Resolution Versions:

Green Amber Red (Traffic Light) Version

White to Red (Colourblind-Safe) Version

Guardian Article

Coronavirus Media Roundup week ended 27 September 2020

Sky News including my cases heatmap analysis at 2:30

Channel 5 News

Quoted in the British Medical Journal
(BMJ 2020; 370 :m3678 doi: https://doi.org/10.1136/bmj.m3678 )

Quoted in The Observer / The Guardian



BBC Local Radio

And interviews on BBC Look North & BBC East Midlands Today; Background to BBC News, Reuters

Coronavirus Media Roundup week ended 20 September 2020

ITV Good Morning Britain

Interviewed on BBC Five Live

Sky News

Analysis shown on ITV Peston

Daily Mail (Mail Online) and Yahoo! News UK

BBC Local Radio (BBC Radio Leicester, BBC Radio Oxford, BBC Radio Berkshire)

and background interviews for BBC, Bloomberg, Reuters.