Written evidence submitted by Prof Ray Pawson (CLL0025)

Biographical details:

Emeritus Professor of Social Research Methodology, University of Leeds.

Author of several books and many papers on evidence-based policy.

Submission: A complex systems approach to crisis management.

Despite a mountain of epidemiological research, the policy response to Covid-19 has remained fragile. This contribution begins by drawing out some key lessons and then provides an analysis of the underlying complexities.

Lessons Learned

• Always prefer real-time data over model scenarios. Use data from a wide range of sources collected using a wide range of methodologies. Complexity theory and systems thinking are crucial and have been conspicuously absent in SAGE advice.
• The more elaborate the scheme, the more likely it is to encounter unconsidered hurdles, unintended consequences, and system failure. Avoid test-and-trace moonshots, five-tier alert systems, multiple reporting systems, and so on.
• The policy response needs to tread carefully between system stability and system adaptation. The main national policy on distancing and closures requires a constant press. Persistent changes in rubric and rhetoric only fatigue practitioners and muddle the public.
• Policy adaptation should be local rather than national or regional. There should be far more emphasis on containing the most harmful micro-circuits of transmission – as in hospitals, care homes, key workers, black and Asian communities, student residences, sweat shops, the evening city centre economy, etc.
• Data, especially national data, never speak for themselves. Beware of preachifying for particular action plans, proffered by powerful lobbies. Research can be disappointingly partisan but good science depends on a ‘disputatious community of truth-seekers’.

Summary

Whilst the world holds on to the promise of an effective vaccine and better clinical treatments, the first response to the pandemic has taken the form of what epidemiologists are inclined to term ‘non-pharmaceutical interventions’ (NPIs). This is a curious designation for it tells us what the response is NOT. And what the coronavirus response actually consists of is a massive array of public policies and social interventions. It is an unprecedented attempt to encourage and instil social and behavioural change on a grand scale. Such complex policy reforms are notoriously difficult to implement and evaluate. Parliamentarians will not need reminding of the decade of toil and trouble involved in shaping and reshaping the Universal Credit system.

This brings us to the question of how research can help in the design, execution, and assessment of such programmes. The first principle is to understand the nature of the beast, to grasp the complexity of the arduous journey from the ideas and inspirations of policymakers to the hearts and minds of the public. Herewith, a summary of those manoeuvres. The policy response …

1. … Consists of an ever-mutating bundle of fragmented interventions (hand hygiene, protective equipment, closure of shops, stadiums and schools, social distancing, rule of six, working from home, testing and tracing, lockdown 1 and 2, etc.) which interact and may complement but may stymie one another. Job retention schemes reduce transmission but drive deficits. Discharging elderly hospital patients without testing increases care home transmission. Isolation measures increase mental health problems, domestic abuse, educational disadvantage but decrease A&E loads, cancer referrals, pollution levels. Vaccine announcements induce complacency, and so on.
2. … Involves long implementation chains, which adapt the interventions on their way to the public. Central government or ‘top down’ interventions often come with pages and pages of guidance, which are reinvented continually by intermediaries over time, generating intended and unintended consequences. Hospitals and care homes extemporise under demand pressures, PPE shortages and staff absences. Policing policy on unofficial gatherings varies by constabulary. Schools differ in maintaining provision for the at-risk and the children of key workers. Parents disagree about safety levels on reopening. Family’s resolve to isolate and distance weakens over time, and so on.
3. … Requires as much if not more attention to ‘exit’ as it does to ‘entry’. Unlocking is significantly more difficult to phase, manage and implement than lockdown. Closing schools, shops, stadiums and so on is much simpler than reopening them with capacity limitations, one-way systems, sanitising points, screening and booking systems. Messaging to ‘stay at home’ is more easily comprehended and actioned than later alerts advising people to ‘stay safe’, and so on.
4. … Is deeply contextual, with the same measures generating different outcomes in different communities and countries. Both the transmission potential and the capacity to respond vary significantly from location to location. Disease prevalence varies significantly by subgroup. The R number varies sharply from neighbourhood to neighbourhood. Compliance with guidance varies with national and local culture. Public health discipline changes. Very young children, dementia sufferers and the drunk and disorderly have little capacity to obey distancing rules. Guidance is continually tested by ‘free riders’ and so on.
5. … Is continually buffeted by political dogfights, with frequent changes in strategy and in action plans. The timing of the introduction and withdrawal of specific interventions is influenced almost daily by media and social media exhortations. The UK response has twisted and turned under pressure from powerful libertarian and authoritarian interest groups (and footballers!). Most significantly, the content and substance of the response has been shaped and reshaped under negotiation with professional bodies, with local authority leaders, with the different national jurisdictions and so on.
6. … Consists of a complex, adaptive, self-transformative system, thrust into a complex, adaptive, self-transformative system. All policies, including those directed at epidemics, operate in a wider cycle of reforms. Coronavirus interventions shape and are shaped by other contemporaneous social movements and political agendas. The virus response impacts upon the ‘levelling-up’ agenda, Brexit negotiation, green renewal, the Black Lives Matter movement, and so on. In turn, all of these reforms have influenced the way in which COVID-19 interventions are implemented.
7. … Uses social interventions to break biological chains of transmission that are often unknown and largely invisible. Estimates for the percentage of people testing positive for SARS-CoV-2 who may be asymptomatic vary wildly (between 5% and 80%). Interventions such as symptom-based screening will inevitably miss cases. The virus, moreover, is a syndrome consisting of several mutating strains which vary in their origin, prevalence and stability. Over time these may generate novel and unanticipated threats.

Complexity abounds. What are the methodological implications? What limitations are placed on ‘the science’? There are two important consequences. I) Causal attribution is extremely taxing, II) Modelling the epidemic’s future is doomed to fail.

I. Causal attribution is the classic aim of applied science. It tackles the ‘what works?’ question. Put simply:

Have the assembled interventions succeeded in controlling the virus?

If the above analysis is correct, the causal question is transformed thus:

Has the interlocking and changing stockpile of adaptive, self-transforming interventions, each one with complex and sometimes contested guidance on its remit, as implemented and switched on and off by a changing array of central, local, private agencies, as digested by a diverse population containing people who comply, resist, learn, grow weary, change their minds, and seek exceptions, succeeded in controlling the virus?

Traditionally, causal analysis in non-experimental (i.e. population) studies operates by tracking the progress of an intervention against its outcomes (prevalence, deaths, etc). This can work reasonably well when intervention and outcomes are simple and singular. But when the intervention is a continually mutating conglomeration, the task becomes much more difficult. All we can say, is what is apparent at the time of writing (11.11.2020) – namely, that collectively the first broad configuration of interventions met with some success in controlling the virus, progress which then faltered as some of the components were removed, which prompted the assembly of a new package of interventions to combat wave 2, which may or may not go on to control the virus. We can chart the broad trends but what we cannot do, because the interventions are mutating, quarrelsome blobs, is to perform ‘contribution analysis’. We cannot pinpoint the crucial causal agents and their specific effects. We cannot say that ‘handwashing to the tune of happy birthday’ contributed to x% of the reduction in transmission, that ‘school closure’ accounted for y%, that the ‘rule of six’ generated z% improvement and so on.

In this respect it is also important to challenge the casual use of language which says that ‘lockdowns’ work. Lockdowns, firebreaks and so on are agglomerations (blobs of blobs). Lockdown 2 will not be the same as lockdown 1, nor is it the same as the lockdowns that have been tried across the world. To be sure, total lockdown (Wuhan style) might well work but only when applied unsparingly in a highly regimented and compliant population. The task in the UK remains that of tailoring and balancing lockdown’s many components in order to anticipate the changing response from its many publics. Trial-and-error is inevitable. Steady progress will follow from multiple, small gains in containing the most harmful micro-circuits of transmission.

II. Much of the early scientific advice on controlling the epidemic came from mathematical biology. Modellers figure significantly in the main SAGE committee, with a couple of dozen more on SPI-M. The modeller’s task is summarised in this figure.

Estimates are produced on the difference between ‘doing nothing’ (cases without protective measures) and ‘doing something’ (namely, the battery of interventions that might put in place). The crucial point is that these graphs are indeed estimates (or scenarios). The equations behind the progression curves are loaded with statistical assumptions about how people will behave under the different regimes. The committee will be aware of the OSR censure about treating scenarios as evidence and about the need to make the underlying assumptions more transparent. It is indeed the case that these crucial assumptions drive the models are often opaque and tucked away in footnotes and report annexes. But the real problem is that they are entirely simplistic.

The actual response to the virus programmes is captured in the intricate manoeuvres outlined in the seven bullet points above. That response, to repeat, is always in flux, is full of intended and unintended consequences, being shaped by the scores of influences and agencies as listed. No model can begin to imitate its complexities. Rather, the estimates driving the models come in the form dictated by the statistical apparatus. ‘Parameterization’, to use the jargon, thus tends to the fixed and mechanical - x% of population will comply with measure A; y% of hospital patients will require ICU treatment, z% of schools will close and so on. In a previous paper, I have scrutinised key assumptions from key models in great detail [1]. I provide a single example here. The influential report 9 of the Imperial College team [2], inserted the following estimate into their model. ‘30% of cases requiring hospitalisation will require critical care’. In model world, this leads to the alarming scenario that ICU provision will be rapidly overwhelmed. In the real world, real-time evidence revealed that the actual throughput was significantly smaller. The guesswork proved faulty and guesswork permeates these models. The fundamental principle, however, is that society’s response to research is always active rather than passive. Under the forecast of a surge in capacity requirements, the very first thing practitioners do is modify and improve local provision. Rather than predicting it, the model prompted behavioural and organisational change.

This brings us back to the core principle of this submission – social and behavioural change under virus interventions is complex, adaptive and self-transformative. Systems-based research methods [3, 4] should be at the forefront of the research effort and this would help sustain a more measured approach to crisis management.

References

[1] Pawson R. The Coronavirus Response: Boxed-in by Models. Evaluation Nov 5, 2020. Open Access. https://doi.org/10.1177/1356389020968579

[2] Ferguson N, et al. (2020) Report 9: Impact of non-pharmaceutical interventions (NPIs) to reduce COVID-19 mortality and healthcare demand. Available at: https:// www.imperial.ac.uk/media/imperial-college/medicine/sph/ide/gida-fellowships/Imperial-College-COVID19-NPI-modelling-16-03-2020.pdf

[3] Gates E (2016) Making sense of the emerging conversation in evaluation about systems thinking and complexity science. Evaluation and Program Planning 59: 62–73.

[4] Gerrits L and Verweij S (2015) Taking stock of complexity in evaluation: A discussion of three recent publications. Evaluation 21(4): 481–491.

Nov 2020