Dr Jonathan Rougier – Written evidence (RSK0004)

 

     Introduction

 

My name is Dr Jonathan Rougier.  I am an applied statistician now working as an independent scientist. Until 2019 I was Professor of Statistical Science and Head of Statistics in the School of Mathematics, University of Bristol.  I am familiar with the UK National Risk Assessment (NRA), as I was seconded to the Civil Contingencies Secretariat of the Cabinet Office, Nov 2016 to Mar 2017, with funding from the Natural Environment Research Council.  My role was to contribute to the final stages of the 2016 NRA (which came out in 2017) and the initial stages of the 2018 NRA.  I am contributing this evidence in a personal capacity.

  1. I will make some introductory remarks, and then focus on the NRA in the sections below.  I will examine the process of the NRA, rather than pick out individual hazards.  I cover several of the questions in the Call for Evidence, notably 3, 4, 6, 9, and 10.
  2. Question 4 in the Call for Evidence asks about the appropriate level of assurance for the UK’s hazard resilience.  I do not think it is feasible to provide any explicit assurance about such a complex topic.  However, we derive implicit assurance from a transparent process for formulating and implementing resilience policy.  This transparency allows for auditing and external challenge.  As such, a transparent process should also be flexible.  In the sections below I outline some suggestions to ensure that the NRA is transparent and flexible.  These suggestions represent a radical simplification of the NRA, at least insofar as it supports resilience policy, which I take to be its objective.
  3. Regarding planning, it is helpful to make a distinction between pre-emptive planning for national-scale events, and responsive planning during the unfolding of such an event. Confusingly, the same term, Reasonable Worst Case (RWC), is used for planning in both contexts.  In pre-emptive planning, the RWC is a pessimistic instance from a specified hazard class, such as ‘Icelandic M6 explosive volcanic eruption’. In responsive planning, the RWC is a pessimistic continuation of the current situation; for Covid-19 at the time of writing that might be a long cold spell in Jan and Feb.  The Call for Evidence covers both types of planning but I will focus on the former.
  4. The Call for Evidence makes the distinction between hazards (non-malicious) and threats (malicious).  This seems to be a Whitehall and Security Service distinction, and it is not recognised in the wider field of risk assessment and risk management.  I find it helpful to use a classification similar to the Sendai (2015) Framework, of (A) Natural hazards, (B) Biological hazards, (C) Industrial and technological hazards, and (D) Other hazards, including malicious actions.  Such classifications are not essential but they are useful for structuring the overall NRA.  From the point of view of risk assessment, I reject the idea that malicious actions are somehow different from non-malicious hazards.
  5. I have one comment about the current Covid-19 crisis.  Statisticians and psychologists have developed a good understanding of people’s biases when assessing uncertainty, such as over-confidence, and also about the biases arising from group dynamics, especially in face-to-face discussions, known as ‘group-think’.  There are simple practices to mitigate these biases.  Usually these practices would be implemented by a Facilitator who is experienced at expert elicitation.  I hope that all reasonable steps have been taken to mitigate biases in the uncertainty assessment and group discussions that have informed Covid-19 policy.  I suggest that the Committee to investigate this, because it could lead to a simple and effective improvement on current practice.

The National Risk Assessment

  1. I will make the case that the current UK National Risk Assessment (NRA) is a good platform for developing policy designed to increase the UK’s resilience to national-scale events, although I also have some suggestions for changes.  The NRA’s advantages are that it is conceptually simple and intuitive.  My suggestions below are to improve its practical simplicity, transparency, flexibility, and cost-effectiveness.  I would be happy to discuss the suggestions in more detail; undoubtedly they could use some refinement.
  2. Let me summarise my understanding of how the NRA currently works; note that I am not distinguishing between the NRA and its more recent implementation, the National Security Risk Assessment (NSRA).  First, a list of hazard classes is drawn up; for example, ‘explosive volcanic eruption’ would be a hazard class.  For each hazard class, a Reasonable Worst Case (RWC) is proposed; for example, ‘M6 eruption in Iceland’ might be a RWC for the hazard class ‘explosive volcanic eruption’.  This RWC is quantified in terms of its Impact and its Likelihood. The former is assessed according to several dimensions of loss (including loss of life, injury, and financial loss), and the latter is assessed as an Annual Exceedance Probability (AEP).  An AEP is the probability of at least one event at least as bad as the RWC occurring in the next year.
  3. At the second stage, each hazard class is represented as a point on a Risk Matrix, with Likelihood on the horizontal axis and Impact, summarised in some fashion, on the vertical axis.  The hazards classes above the diagonal of the Risk Matrix towards the top right-hand corner have high risks (I am simplifying slightly for brevity).  The principle of the the NRA is that UK policy should be directed towards building resilience to the consequences of the high-risk hazard classes.  It is important to appreciate that resilience is not to one particular event or hazard class, but to the consequences of the high-risk hazard classes.  The idea is that by building resilience across these consequences, we also build resilience to the consequences of other hazard classes, both those further down the list, and those which were not foreseen (‘unknown unknowns’).
  4. One hears the phrase ‘cluster/cascade of risks’ (including from the Committee Chair, Q32 in the meeting with Professor Sir John Beddington and Professor Sir Mark Walport, 9 Dec 2020, uncorrected transcript).  To clarify, it is typically not the hazard events themselves that cluster or cascade.  It is the consequences that cluster, because one event can have multiple consequences, and it is the consequences that cascade, because in combination they can have unforeseen and undefended further effects.  Clusters and cascades increase risk.  ‘Space weather’ is a high-risk hazard class because a large solar storm has several consequences, and some of these, such as loss of power and loss of Low Earth Orbit satellite functionality, might cascade into other consequences, such as failure of supply chains and financial markets.  The NRA’s focus on the consequences of the high-risk hazard classes should automatically account for consequence clusters and cascades, and is one of its strengths.
  5. Overall, the NRA as I have summarised it is a conceptually simple and intuitive approach to risk assessment for national-scale events, and a launchpad for resilience policy, notwithstanding that there are unresolved issues about choosing the RWC for each hazard class, and quantifying its Impact and Likelihood.  I will come to these in the next section.
  6. Finally, there are some complex topics that are not well-suited to the NRA, as the NRA itself recognises.  These play out over a longer time-period, and are typically not event-driven, although they give rise to events.  Two well-known examples are climate change and antimicrobial resistance.  The NRA is a useful framework because it covers a large number of hazard classes, and is extendible to many new hazard classes.  Complex topics will need a more bespoke treatment, as represented by the Climate Change Committee, which was established under the Climate Change Act (2008), and the 5 year Action Plan for Antimicrobial Resistance 2019-2024.

Some suggestions for the NRA

  1. We must recognise at the outset that risk assessment for national-scale events is very challenging.  Where numerical summaries are required, for the purposes of ranking and resource allocation, these summaries will be highly uncertain; some authors would call them ‘radically uncertain’.  Nevertheless, we must to try to quantify, because the list of possible national-scale events is long, and we cannot produce a bespoke policy for every one.  But we should start by accepting that our quantifications of Impacts and Likelihoods are only accurate to within a factor of 10, say.  This will still be sufficient to identify the high-risk hazard classes.  Once we accept this inherent inaccuracy, we can dramatically simplify the process of risk assessment for each hazard class.
  2. For a given hazard class, my first suggestion is that we should use the worst relevant event in the historical record as the RWC.  This sidesteps ‘What do we mean by RWC?’ in favour of ‘What is relevant to the UK?’, which is much easier to address.  Sometimes ‘relevant’ will simply be events which have happened in the UK.  But there will be other situations where the pool could be widened to include events which happened elsewhere but which might have happened in the UK (examples are given below).  This suggestion also makes it easier to assess the consequences and Impact of the RWC, which can be based on historical records, and then rebased to modern values.  To clarify, by ‘historical period’ I mean the period of reliable written records.  For some hazard classes, such as explosive volcanic eruptions, the geological record is much longer.  However, this record gives little detail about consequences and Impact.
  3. My second suggestion is to represent the Impact of the RWC entirely in monetary terms. This  means providing conversion rates for death and injury, something which is already done explicitly in Health and Transport policy, and done implicitly through the Courts in compensation claims.  There are some dimensions of Impact that are hard to monetize, notably loss of some services, and psychological harm.  It is important to appreciate that not quantifying these dimensions does not mean that they do not count.  Instead, they are counted implicitly under the assumption that they scale linearly with the monetized dimensions.  This assumption misses the ‘dread factor’, which is that a death from, say, radiation poisoning generates more public concern than a death from the flu.  A ‘dread factor’ could be included in the Impact assessment, but many people would question its presence in resilience policy.
  4. This leaves the assessment of the AEP of the RWC, the Likelihood.  My third suggestion is to use the reciprocal of the length of the historical record, if the RWC has been chosen according to my first suggestion.  Expert judgement might raise or lower the Likelihood by factors of 10, to bring the historical record into line with the UK next year.  These adjustments will account for changes of scale, and also for non-stationarity, which is where next year in the UK is not similar to the average year in the historical record.  Examples are given below (Paragraphs 17 and 22).
  5. Take, for example, the hazard class ‘vehicle ramming attack against pedestrians’.  If we take the record since the year 2000, the worst such attack was Nice in 2016, when 86 people were killed and 458 injured: take this as the RWC.  So the Impact of this hazard class would be 86 people killed and 458 injured.  It would be natural to lower the Likelihood by a factor of 10, from 1/21 to 1/210, because the UK is only a part of the pool of relevant countries.  Experts might want to lower it further, given the enhanced monitoring activities of the Security Service, and the counter-measures now emplaced.  But there is little point in agonising over this, because even at a Likelihood of 1/21, ‘vehicle ramming attack’ is not going to be among the UK’s high-risk hazard classes.  I stress that I am not an expert on terrorist risk: this is merely an illustration that a simple framework for assessing the RWC and its Impact and Likelihood can apply across all hazard classes, including ‘threats’.
  6. These three simplifying suggestions for the NRA will strike many people as crude, especially when there is so much expertise available to provide more nuance.  But I would respond that they are sufficient for the NRA, where we mainly need to identify the high-risk hazard classes.  There is definitely a requirement for expertise in resilience policy, but it mainly comes after the compilation of the NRA.  Once we have identified the high-risk hazard classes, we need a careful narration of how national-scale events in each hazard class might unfold, what the consequences are, how they cluster and cascade, and so on.  And then we need expertise to formulate and implement resilience policy to mitigate the consequences in a cost-effective manner.  These are where the major challenges lie.  Expertise spent on the NRA is being allocated to the ‘easy’ part of resilience policy, when we should be saving it for the challenging part.
  7. There are four further features of this simplified NRA that I would like to highlight.  First, it can be based entirely on publicly available information.  This enhances transparency, because the entire assessment can be placed in the public domain, unlike at present. 
  8. Second, consistency across hazard classes is ‘built-in’, owing to simple guidelines for deciding the RWC, and its Impact and Likelihood, which are based on publicly-available information and which hold across all hazard classes (almost all, see Paragraph 21).  In his evidence to the Committee (meeting of 25 Nov 2020, Q7, uncorrected transcript), Roger Hargreaves, the current Director of the Civil Contingencies Secretariat (CCS), recognises that consistency across hazard classes, and the communities which study them, is an issue in the current NRA, requiring mediation by the CCS.  I think it would be better if the consistency was built-in.
  9. Third, I do not attempt to be comprehensive.  I would not delve into hazard classes where there are no national-scale events in the historical record.  And, as the ‘vehicle ramming attack’ shows (Paragraph 17), I would be quite perfunctory with hazard classes that are very unlikely to be high risk.  The objective of the NRA is to support resilience policy, and I agree with the principle that we can do this effectively by planning for the consequences of the high-risk hazard classes that we currently identify.
  10. Fourth, I do not assume compliance with current regulations.  There will be pressure on experts to assume compliance when assessing Likelihood, particularly if they are insiders, as they often are. The disaster at Grenfell Tower illustrates that we should not assume compliance in risk assessment.  Under my suggestions, Grenfell Tower would become the RWC for the hazard class ‘urban fire’, and the Likelihood might be the reciprocal of the number of ‘tower-block years’ in the UK.  In due course this might be reduced by a factor of 10, as compliance improves.  Similarly, the RWC for the hazard class ‘major accident at a nuclear site’ would be Chernobyl.  Under compliance with the Office for Nuclear Regulation (ONR) the AEP for a major accident at a nuclear site should be lower than 1/100,000.  But under my suggestions the Likelihood might be the reciprocal of the number of ‘reactor years’ in countries with substantial nuclear power programmes, times the number of reactors in the UK.  And then possibly reduced by a factor of 10 on the basis that current compliance at UK sites is far better than average compliance over the historical period.  Again, I am not an expert in either of these two hazard classes; I am using them to demonstrate that we do not need to assume compliance.
  11. My suggestions are for an evidence-based NRA which can exist entirely in the public domain, with the minimum of expert maintenance.  This makes it suitable to be an online resource, rather than the current biennial report.  Not only does this substantially enhance transparency, but it also enhances flexibility.  Very occasionally, an event will eclipse the RWC in its hazard class, and then the webpage for that hazard class will need to be revised.  New hazard classes might arise.  Otherwise the ongoing changes will be minor.  Impacts, measured in pounds (notional), will need to be revalued for demographics, and possibly also for changes in the conversion rates. As the years tick by and the historical period extends, the Likelihoods will need to be reduced.  Thus the NRA becomes an online and easily-updated resource.
  12. In summary, the conditions that shape my suggestions for the NRA are: (i) there is ‘radical uncertainty’ in risk assessment for national-scale events; but (ii) our task is really quite simple, to identify the UK’s high-risk hazard classes, so that we can develop resilience policy for their consequences. Under these conditions, we do not need a gold-standard process which consumes large amounts of time in a two-year cycle to produce a thick report.  Instead, I suggest we can use a simplified approach which builds on the conceptual simplicity and intuitiveness of the current NRA, and enhances its transparency and flexibility.  My suggestions are cost-effective, and the resources saved can be redirected into resilience policy, which is where the major challenges lie.
  13. I should also clarify that my suggestions are entirely about the NRA and its use in resilience policy.  There may be other reasons to do a more detailed analysis of the risk of a particular event or hazard class, but that would be a different project and a different budget.

Resilience policy

  1. Formulating and implementing resilience policy is where the major challenges lie, as many more general examples of policy failure testify.  If the CCS adopts my suggestions for a simplified NRA, then they will be able to focus their effort on resilience policy.
  2. The key thing about resilience policy is to recognise that although it is challenging to formulate and implement policy, especially in the Public Sector, there are strategies for avoiding pitfalls and raising the chances of success.  The staff of the CCS need to study the theory and practice of policy.  As a starting-point, I cannot do better than to recommend two book-length treatments.  First, Richard Rumelt, Good Strategy, Bad Strategy (Profile Books, 2011, 322 pages).  In the context of the NRA, I would highlight the discussion that coordination, by itself, can be an important source of advantage (Ch 5). Second, Anthony King and Ivor Crewe, The Blunders of our Governments (Oneworld Publications, 2013, 496 pages).  This book is a salutary reminder of how smart people can create bad policies.  In the context of the NRA, I would highlight the sequence of chapters on ‘Human Errors’ (Chs 16 to 20).
  3. Finally, I draw attention to two resilience policy issues with political implications.  First, spending targeted at specific hazard classes is far more visible than spending targeted at common consequences.  For example, everyone can see a new flood defence on the River Severn, and it provides comfort even when there is no flooding.  But few people will see or draw comfort from a carefully drawn-up evacuation plan, a fuel stockpile, or an emergency morgue.  So, politically speaking, planning for common consequences is a less attractive way to spend money.
  4. Second, what is the role of the State when individuals and organisations suffer losses which they chose not to insure against? Insurance is a form of risk transfer, and an important tool in resilience policy.  But issues concerning the State’s obligations to its citizens have many political ramifications.  In the Covid-19 crisis, the State has compensated individuals who have not taken out insurance against loss of income, and organisations which have not taken out insurance against loss of business.  We must consider how this compensation has redefined the role of the State in people’s minds, and whether it now makes them less inclined to insure.

Brief glossary

  1. The language of risk is confusing: entire articles are written about the meanings of ‘risk’.  I have been careful to use a small number of reserved words in a consistent fashion.  For completeness, these are:

         National-scale event. An event which overwhelms local responders.

         Hazard class. A set of events with a common etiology.  Might also just use ‘hazard’, but this has a separate meaning in the NRA.

         Reasonable Worst Case (RWC). In pre-emptive planning, a pessimistic event from a specified hazard class.

         Consequences. The things that follow from an event, and cause losses. Can cluster and cascade.

         Impact. A simple numerical summary of losses in the RWC.

         Likelihood. The Annual Exceedance Probability (AEP) of the RWC: the probability of at least one event at least as bad as the RWC occurring in the next year.

         Risk. Formally, Impact times Likelihood.  Informally, an ordinal measure of severity, when applied to an event or a hazard class (e.g, ‘more risky’, ‘high risk’).
 

 

Jonathan Rougier
In a personal capacity


31 December 2020