Why were the modellers wrong (again)?

The recent MPs report about how we responded to COVID has generated some media interest in the nature of science and its relationship with policy.  We have commented on the issues raised when we are ‘led by the science’ in a previous blog, but the recent discussion has focussed on the fact that being wrong is part of the scientific process.   And the same principle applies to modelling, in fact those in the modelling world often trot out the much-used statement that ‘all models are wrong, but some are useful’. 

The image below appeared in the advice given to SAGE in early September that then informed the Government Response.  It provides an England wide view on possible levels of hospital admissions during September and October.  WSP provides local modelling at a Hospital Trust catchment level, aggregated to Integrated Care Systems, which normally consist of between 2 and 4 separate hospitals.   We, like the modellers at Imperial, Warwick and the London School of Tropical Medicine & Hygiene were expecting September to see rising numbers of people being admitted to hospital, although our expectations were at the low end of the green wave in the SAGE advice as shown in the second image below.

But we were all wrong!  Between the 7th September and the 9th October instead of seeing steadily rising hospital admissions we saw a reduction from 801 (with a 7-day average of 778) to 685 on the 11th October (and a 7-day average of 635 on the 8th October).  So, instead of either doubling, or rocking into the stratosphere during September, admissions actually fell by about 15-20%. 

Of course, for modellers this represents the comfortable side of being wrong.  Erring on the side of caution, planning for the worst and hoping for the best, are useful ways of proceeding when ignoring the problem and hoping it goes away could result in us being singularly unprepared.  But we still owe people an explanation, and a revised set of modelling lines as soon as that explanation is understood and replicated in the modelling.  So, this is our take on what’s happening, and perhaps what might happen over the remaining part of the winter.

It is evident that significant natural and vaccine immunity has built up in the population.  Were this to be evenly spread across different population groups and ages it would be much easier to model and therefore forecast.  However, the vaccine roll-out has proceeded by age groups and there are additional differences between communities in their experience of the pandemic and therefore levels of antibodies to protect against future rises in cases.  This can create natural ‘fire-breaks’ in transmission either geographically or between age groups, which is almost certainly part of the explanation for not seeing admissions rise during September in the ways expected.  And so, we have a story that might go something like this:

  • In June and July, at a time when younger adults were only part way through having their first vaccine, the country was chomping at the bit to exit from lockdown. 
  • The Euro championships, warmer weather and new freedoms gave many the excuse they needed to mingle – we now know that positive test cases in the under 40’s in early July accounted for around 80% of all positive cases meaning that a firebreak against future spread was being created.
  • When we come to September cases amongst children have rocketed with, for example, the 10-14 age group positive test cases being 5 times the background population levels. 
  • But children are rarely admitted to hospital and the 20-39 age group who could have been instrumental in creating the chain of transmission to other older age groups who were more vulnerable and likely to need hospital care, but were less likely to do so as a result of the June/July peak in cases.

So, it seems that we inadvertently created a perfect calm meaning that instead of rising hospital admissions we have seen, since mid-July, either flat or slightly falling admissions, albeit still at levels that challenge local capacity. 

What happens next?  Well, we’re bound to be wrong of course, but for capacity planning purposes we’re now suggesting that the so-called endemic nature of COVID reflects a finely balanced situation where waning immunity is addressed by booster jabs whilst any pockets of vulnerability due to lower-than-average levels of vaccine or naturally induced immunity could cause local clusters of cases.  These clusters are unlikely to lead to rapid or widespread community transmission but can none-the-less cause temporary increases in cases and admissions at a local level that people need to be aware of.

But there’s a sting in the tail because the level of hospital admissions experienced in many places at the moment, as well as pressures in other parts of the system including General Practice, cannot be explained solely by the normal slowly changing demographic changes plus some COVID.  It appears that there is probably an equivalent level of need being generated today as a result of the impact of COVID on people’s health over the last 18 months as is being generated by new COVID cases.  That, alongside endemic COVID, appears to be the real challenge for the winter.