Coronavirus disease 2019 (COVID-19) modelling has shown itself to be drastically important to public health policy. With millions of deaths and hundreds of millions of cases worldwide, knowing the likely spread and risk of the disease helped governments apply lockdowns, social distancing measures, and other restrictions to minimize the spread of the disease and buy time for vaccine development.

Study: Predicting the unpredictable: how dynamic COVID-19 policies and restrictions challenge model forecasts. Image Credit: Bihlmayer Fotografie/ Shutterstock

During a global health and economic crisis, balancing the potential capacity for opening the economy against risking further transmission was essential. Models can also help select areas for vaccine trials by determining the geographic likelihood of high incidence rates. One particular agent-based model was used to identify the potential rates for the spread of COVID-19 between June and August 2020. Researchers from the Burnet Institute in Melbourne have been using this to investigate how accurate COVID-19 modelling has been.

A preprint version of the study is available on the medRxiv* server while the article undergoes peer review.

The study

The initial model predicts incidence and diagnosis rates for COVID-19 worldwide, examining 72 locations in Australia, Belgium, Brazil, France, Italy, Mexico, The Netherlands, South Africa, Spain, and the United States. It predicted cases for a six-week window during September and October 2020, two-three months from the date of predictions. This six-week window was the time in which the Janssen vaccine would begin clinical trials.

The model used was an agent-based model which simulates decision making entities called agents that base their decisions on a series of rules. In this case, 100,000 agents were used representing individuals who would interact with each other. The model was calibrated for each of the 72 locations using several variables: population size, age structure, household size distribution, contact data through employment, household and school/community contacts, and global disease parameter estimates.

Interventions such as testing, contact tracing, quarantine and other restriction such as social distancing or mask usage could be implemented to alter the model. Five models were initially created – showing no change in restrictions, 10% easing of restrictions (and 10% increase in individual transmission), 20% easing of restrictions and increase in transmission, and a model with no infections imported from outside as well as a model with constant testing. Data was gathered from the simulations once the models were adjusted for accuracy with the current reality in each area.

The researchers examined how often the data fell within the upper and lower bounds initially specified and which conditions were statistically significant in the models success or failure to assess the accuracy of their initial model. Unfortunately, 45/72 projections fell out of bounds and were not accurate.

Generally, the models that were closest to reality had correctly predicted future health policy or testing rates or longer projection periods. Other factors not involved in the prediction that proved to be beneficial for accuracy were a more youthful population or higher unemployment or poverty rates. A younger population could aid more in assuming transmission, as there will be fewer individuals in nursing homes and a generally more resistant population.

As higher deprivation index ratings are correlated with increased transmission and severity, higher poverty and unemployment could also be beneficial in predicting the spread of the disease. The most important factor by far was public health policy assumptions. The most erroneous predictions occurred when the modelling predicted the opposite restriction change to reality, such as loosening restriction when the model showed tightened restrictions. Even when the scientists correctly guessed the direction of the changes, the models were only accurate in 49% of cases.

Conclusion

The authors highlight rapid policy changes from multiple governments as one of the difficulties of modelling any disease, particularly in the COVID-19 pandemic. Particularly in the early days of the pandemic, as well as the initial loosening of restrictions, government advice could change regularly, making accurate predictions difficult. The researchers argue that this supports the need for more revision of modelling and projections as government policies change, as well as revealing the risks of modelling too far into the future. As well as this, the results show the importance of public health policies both in understanding the spread of disease and preventing that spread.

The unpredictable and sometimes unintuitive changes largely explain the failures of COVID-19 models and provide strong motivation for researchers to work with policymakers as much as possible and seek the advice of behavioural experts to determine the reaction to certain restrictions.

*Important notice

medRxiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behaviour, or treated as established information

Written by

Sam Hancock

Sam completed his MSci in Genetics at the University of Nottingham in 2019, fuelled initially by an interest in genetic ageing. As part of his degree, he also investigated the role of rnh genes in originless replication in archaea.