Show Your Working: Model Quality Assurance in Government & Letter in Financial Times

School pupils have had a tough year. For those with examinations, such as A-levels, it has been even tougher. The Department for Education decreed that A-levels and GCSEs should not take place due to the COVID crisis.

This meant that an alternative way of allocating grades to students needed to be found.

To set the context, Ofqual, the Office of Qualifications and Examinations Regulation, regulates qualifications, examinations and assessments in England. It is a Non-Ministerial Government Department constituted under the Apprenticeships, Skills, Children and Learning Act 2009. Now, it is evident that, despite being a non-ministerial department, the Department for Education has a responsibility for setting policy.

Therefore, a Memorandum of Understanding exists between Ofqual and DfE setting out how the two organizations will work together. Section 2 sets out responsibilities:

On 31 March, the Secretary of State, Gavin Williamson, wrote to Ofqual, setting out a ministerial direction. These are the salient points in the Direction:

Firstly, there needs to be a calculation, and secondly ‘as far as possible, the qualification standards are maintained and the distribution of grades follows a similar profile to that in previous years’.

This is a very strong constraint, and to be fair, one that Ofqual met in their initial calculated grade. The problem was it was not fair.

Much debate has been generated as to algorithms, models, and their use in Government (and arm’s length bodies such as Ofqual). But this is not a new problem.

In 2012, the West Coast rail franchise was awarded to FirstGroup. Virgin Trains complained, serious errors were found in the model used by DfT, and the award to FirstGroup was cancelled.

This cost taxpayers £54 million, and, as a result, and report was commissioned to prevent this happening again.

In 2013, Nick Macpherson, Permanent Secretary of the Treasury, published a Review of Quality Assurance of Goverment Analytical Models.

Macpherson’s report was operationalized by HM Government by The Aqua Book: Guidance on producing quality analysis for Government.

The Aqua Book recommends that there should be a Senior Responsible Owner for each model (for Government departments and their arm’s length bodies, such as Ofqual).

Ofqual’s model documentation was set out in their 318-page report Awarding GCSE, AS, A level, advanced extension awards and extended project qualifications in summer 2020: interim report, but the model code has not yet been published.

I cannot find any reference to the Aqua book or ‘quality assurance’ in this model documentation.

Here is my letter about this in the Financial Times


Data Sources for COVID-19 in England – Letter from PHE to Local Authorities

I have received a copy of the following letter from Public Health England to Local Authorities setting out the data sources available to local authories and the public. The letter is dated 9 July 2020 and I have redacted certain information.

This should be useful for people wishing to understand what data is (and was) available on COVID-19 cases in England.


Data Sources for COVID-19 Analysis in England

Since the early Number 10 Downing Street press conferences, the data available to analyze the progression of the COVID-19 epidemic in the UK has become somewhat fragmented. This is an overview of the major sources of data.

This shows data for testing, cases, healthcare, and deaths, updated daily.

More detail is shown on each data set, for example cases:

which is broken down by Nation, Region, Upper Tier Local Authority (‘UTLAs’ / counties) and Lower Tier Local Authorities (‘LTLAs’ / districts). Note that some councils, such as Leicester, are Unitary Authorities, and are both UTLAs and LTLAs and their data is disclosed in both UTLA and LTLA data sets. Currently, this only shows the latest data, and cases per 100,000 people (making it easier to compare different sized locations).

This data is shown as a map at Middle Layer Super Output Area (‘MSOA’) level (although the colour bar makes interpretation a little difficult) and shows cases per week. The mean population of a MSOA is around 7,200 people, so the key is not very useful.

Data for UTLAs and LTLAs used to be reported by specimen date, but it is not clear where these data are now available.

Public Health England Surveillance Reports

Public Health England (‘PHE’) produce a COVID-19 surveillance report each week. This now shows a list of watchlist local authorities.

These data are shown on a

together with a chart of the number of English cases per week

and the history of the highest rated UTLAs

There is also a report of the number of ‘incidents’ (what are colloquially described as outbreaks) and where they originate from.