Public Administration and Constitutional Affairs Committee 

Oral evidence: Transforming the UK’s Statistical Evidence Base , HC 197

Tuesday 6 February 2024

Ordered by the House of Commons to be published on 6 February 2024.

Watch the meeting 

Members present: Mr William Wragg (Chair); Ronnie Cowan; Jo Gideon; Mr David Jones; Lloyd Russell-Moyle.

Questions 151 - 180

Witness

I: Ed Humpherson, Head of the Office for Statistics Regulation.

Written evidence from witnesses:

– Office for Statistics Regulation

Examination of witness

Witness: Ed Humpherson.

Chair: Good morning and welcome to the Public Administration and Constitutional Affairs Committee. Today, the Committee is holding its fourth oral evidence session in our inquiry into transforming the UK’s evidence base. We are exploring how standards differ between official statistics and other forms of analysis, whether officials adequately engage with users of evidence to understand their needs, and the performance of the Office for National Statistics. This morning we are joined by Ed Humpherson, whom I ask to introduce himself more fully for the record.

Ed Humpherson: I am head of the Office for Statistics Regulation, which is the UK’s regulator for official statistics and data.

Q151       Chair: Thank you very much. An open question to begin with, please: in your view, what makes for "good” evidence?

Ed Humpherson: What we look for is evidence that complies with the three pillars of our code of practice: trustworthiness, quality and value. So we look for evidence that is produced in a trustworthy way, with orderly release and a clear separation of the policy interest from the analytical expertise; we look for quality in the sense that the data are understood, the methodologies are sound and the quality is assured; and we look for what we call value—in other words, that it meets users’ needs, provides insight and answers users’ questions. We regard those three pillars as pretty universal. Obviously, we apply them to official statistics, but in our view—and we implement this view across our work—they apply much more broadly to other types of evidence as well. It is a very flexible framework.

Q152       Chair: We have heard from a range of witnesses, and while standards are generally high in the field of statistics, they can sometimes fall short in the broader suite of analysis that takes place across Government. Has that been your experience?

Ed Humpherson: I have been reflecting on the evidence you have received on this, and it seems to me that there is a difference of perspective between people inside and outside Government. The perspective of analysts inside Government is that there is quite a good picture about the UK’s evidence base: we have good structures, we have a strong analysis function and we have a digital and data office. There is a very good evaluation taskforce. The leadership of the civil service is saying really empowering and important things about the role of data and analysis and there are new tools and techniques; data science is becoming increasingly common across Departments. This system, from inside, produces some really good work and I can run off a litany of powerful and valuable pieces of analysis, like the census visualisation, the BOLD programme—Better Outcomes through Linked Data—and the levelling-up work. There is lots of good stuff.

On the other hand, from the perspective outside Government, the picture is much patchier and I think the differentiation is not so much the strength of the evidence base but the way in which it is put into the hands of users in the public. If that evidence is official statistics, complying with our code and regulated by us, it comes out in a very clear and structured way, and it is available. Other types of evidence, such as modelling—the kind of evidence that is done to understand the operations of Government—have a less uniform way of getting into the public domain. As a result I think that users, such as Parliament, academics and the media, have a much less complete picture of the evidence base than people inside Government. That is a theme of what I want to say: that there are ways of making the whole evidence base much more legible and transparent to outside users.

Q153       Chair: Understood. It is fair to say that you think some bodies do analysis better than others, and you have named a few good examples. Which are the ones you think, if I can put it this way, need to most improve?

Ed Humpherson: I do not look at it quite in terms of different entities of government; I look at it in terms of different purposes. One purpose of analysis, for example, is to understand the fiscal position of government. In the UK we have really strong fiscal forecasting; we have the Office for Budget Responsibility and its comparative bodies in the devolved Administrations, like the Scottish Fiscal Commission. However, if you look at evaluation, which is after the fact—looking back on policy—the record about commissioning evaluation and then making that evidence available is much less complete.

I would not say it is an issue of individual Departments being better than each other: it is individual purposes being better served, and evaluation is probably one of the weaker areas and one thing we would really push for there being more transparency about.

Q154       Chair: Could you perhaps give the Committee an example of a policy or some initiative whereby evaluation has been non-existent or not acceptable, and of how better evaluation can improve things in the future?

Ed Humpherson: Can I give you an example that ended up being a happy story, regarding the troubled families programme? Brought in with great fanfare, the Government commissioned an evaluation and undertook that evaluation, but then it took some time for it to be released and, as a result of that delay, external users suspected it contained a bad news story. Under the drive of the evaluation taskforce—this new taskforce I have mentioned already—it was eventually put out into the public domain, and it was a really strong and effective evaluation.

It is not therefore bad evaluation that I would point to but the availability, accessibility and transparency of evaluation. The troubled families story says that actually, when it is made available, it tends to show a reasonable and strong picture of Government. Of course, there will be some evaluations done which show that policies have not worked as intended, but as long as the Government are willing to make the bad news available, people will be confident in the good news when evaluations show there is good news.

Q155       Chair: Yes, in the course of this inquiry we have found that some official analysis never quite sees the light of day, and we try not to be suspicious and sceptical as a Committee, but you can understand why others might, as you have said, become suspicious. Are you monitoring any analytical capability that has been undertaken across Government, beyond statistics?

Ed Humpherson: Not in a formal way. We have our hands full upholding the code of practice for official statistics. We would be very happy to extend our remit to undertake assessments of analytical capability more broadly, but to do that we would need a remit, and we would, frankly, need resource as well.

Q156       Chair: Has the OSR, in its history, done that before? Or is it something it has not done but might wish to do?

Ed Humpherson: We have done two things. First, we have looked at statistical and analytical leadership—in fact we have a report on analytical leadership coming out this spring, in a couple of months’ time—so we have looked at the issue generically. What we have not done within that is delve down into individual parts of Government to form a view of capability.

The second thing we have done, and I think this is one of our real triumphs, is that we have established this principle that we call intelligent transparency, which is that when a Department or a Minister makes a statement involving quantitative analysis—numbers—the underlying analysis should be made available to all at the time the statement is made. If it is not, we step in, and we always do so by re-emphasising and reiterating those principles of intelligent transparency.

I think of this as being like a garden on a bright morning where the shadow is gradually retreating and the sunlight is coming across. We have the sunlight on official statistics, we are gradually moving it into that space of Government statements being supported, and I would love that sunlight to extend into further parts of analysis as well.

Chair: Thank you very much.

Q157       Mr David Jones: During this inquiry, we have heard several times that decision makers are turning to new, real-time data sources, which apparently have great potential. Is there a risk if decisions are made on the basis of data which are not being assessed for quality and trustworthiness in the same way that official statistics are? Could you comment on what this means for policy development?

Ed Humpherson: The first thing is that you are quite right that there is a risk. These new data sources have many attractions, as you have heard from previous witnesses, but they also, like any data source, have their limitations. One of the limitations of real-time, non-survey-based sources is that the analyst may not have a complete understanding of the biases within the data set. It is really important that when those data are used, there is an understanding of whether there are subpopulations, subgroups and subcategories that are not covered; otherwise, they may draw conclusions across the whole population that are not warranted because they are missing the subgroup.

You are absolutely right about the risk and this is why, going back to my very first answer, we would encourage all Government Departments to deploy our pillars of trustworthiness, quality and value right across their data estate, not simply for the things that are formally within our remit of official statistics.

Q158       Mr David Jones: How safe is it to develop policy on the basis of such sources?

Ed Humpherson: Well, by talking about risks I may myself be running the risk of painting too alarmist a picture. There is a risk, but it is totally manageable by good-quality analysis that understands the limitations of the data.

In the examples that I have seen of the use of more real-time data sources, such as by the ONS during the pandemic with real-time economic indicators, the risks were well explained, well understood and well managed. The danger is when analysis is done without consideration of those risks, so you can have a situation where too much weight is put on the data source. But in the examples I have seen, the risks have been well managed.

Q159       Mr David Jones: Does the OSR have the necessary tools to enable it to assess products other than statistics—for example, ad hoc releases or models produced by Departments? These may well be necessary, but if they are going to be relied upon by Ministers, they do need that assurance we have been discussing.

Ed Humpherson: I would draw a distinction here between the technical component of a piece of modelling, the professional skills that go into that technical component, and the principles I have been talking about of appropriate release with clear caveats and so on.

What we know about as OSR is how to get information of a quantitative nature into the public domain in a way that minimises the risks of being misleading. Our tools across the board are really good for any piece of quantitative analysis to say, “This is how you should make it available publicly: you should explain your caveats, engage with your users and so on.”

Where I would not over-claim what we can do is to form judgments upstream—so as to whether an operational researcher has used the appropriate modelling algorithm or whether an economist has got the right econometric analysis. That is not our core skillset. There are professional standards across Government that are sure on those things, and I would not want to expand my empire so far as to encompass that technical side. What we would like to effect, and what I would really encourage you as a Committee to emphasise when you report, is a common framework for making all these different sources of quantitative information available to users and public audiences in a consistent way. That is where we add our value in this space.

Q160       Mr David Jones: Some witnesses to the inquiry, as you probably know, have suggested that the OSR should be resourced to monitor numbers beyond official statistics. How do you feel about that recommendation?

Ed Humpherson: I do not resist it, for sure, because, as you can tell, we are champions of having a credible evidence base in the hands of the public. If your witnesses, and you as a Committee, think we can usefully play a role in supporting that by doing a wider remit of work, we would be very happy to take that on, appropriately resourced. But I would want to manage everybody’s expectations, including, frankly, the analysts in Government. I am not pitching for a role where we become the all-purpose checker of all internal pieces of analysis—that would be beyond our scope— but certainly, if there is a model or a piece of policy analysis that is supporting policy, we think there should be clear standards about how that is made available to accompany policy statements.

Q161       Ronnie Cowan: There are a couple of questions that come under the subtitle of “Opening up access to evidence”; I will give you a brief background. In this Committee’s recent report on planning for the future of Government estates, Members found that the Cabinet Office made several high-profile statements about the economic benefits to be delivered by moving posts to the new regional offices, without publishing the underlying research to support its estimates. In a submission to this inquiry, Full Fact provided several examples of Home Office Ministers citing unpublished operational data which cannot be reconciled with published official statistics. The OSR tried to address this in 2022 in new guidance that encouraged the Government to proactively publish evidence where that evidence was being cited publicly. Has that guidance achieved what you hoped it would?

Ed Humpherson: In part. Clearly, the Full Fact and the Cabinet Office examples that you raise are disappointing examples of where this has not been followed. Actually, I had not been aware of the Cabinet Office examples, but I am now—thank you, we will have a look at that. The core principle is: if an assertion is made publicly on the basis of data, those data should be equally available to all.

On the Full Fact examples, we have worked with Full Fact on them, and we have used our leverage, particularly with the Home Office but also with the Foreign, Commonwealth and Development Office on one of the cases, to obtain and require the Department to make underlying data available.

The Home Office story shows the impact that we can have because over time the analytical leaders in that Department have increasingly embedded the standard expectation that numbers are made available into the Home Office’s processes, and they are, largely speaking, complying with that. Of course, people raised with us the situations where that did not happen, but over time you have seen real improvement in the discipline within the Home Office, and I commend the chief scientific adviser and director of analysis for what they have achieved there.

Q162       Ronnie Cowan: You said “In part”; is that a good example?

Ed Humpherson: The Home Office would be the good example. I am concerned that whenever we talk to senior officials about this—we call it intelligent transparency—we get sign-up: senior officials and permanent secretaries endorse and recognise it. We then find cases where, in a specific moment, a particular Department concludes that it is not in its communications interests, or it forgets to make the underlying data available. I would really want to see those commitments that we hear from senior officials being made much more publicly and much more embedded into their practices and processes. It should not happen that there is some really good analysis that supports a policy statement, the policy statement comes out with numbers and you as a Member of Parliament, the media or a member of the public cannot find the numbers. It is a simple hygiene thing.

Q163       Ronnie Cowan: I find it difficult to believe they just forget, but we will move on from that. What can you do to ensure best practice?

Ed Humpherson: We can continue to do what we do which is to advocate these principles, as I hope I am advocating to you. We can step in when there are problems, and we can raise the problems publicly. A useful additional tool in our tool bag would be for us, when we have repeated concerns in a particular area, to raise it with you as a Committee, as a Committee that cares about evidence in Government. I would be very happy to do that in future if that were something you would welcome.

Q164       Ronnie Cowan: You said you had to step in; on several occasions you have needed to reprimand Departments where they have failed to publish data referenced by Ministers in public. Do Departments always publish evidence after you have publicly called on them to do so?

Ed Humpherson: Yes—sometimes with a bit of a lag, but yes. When we step in and say, “You’ve not made this available,” we will work with the chief statistician in the Department, the chief scientific adviser and the director of analysis—those three crucial roles—and we will highlight the issue. They will almost always say, “Yes, we see the point. Leave it with us,” and they will work with their comms teams to get things published, which is what happened in the Home Office cases that you are referring to. We do secure that, but sometimes it could be done a bit more quickly, and I would prefer not to have to step in on this at all. I would prefer it to be the norm.

Q165       Chair: Ronnie, might I ask a brief question on that point? Sorry to interrupt you. On those examples, does the data back up what the Minister has said? If it does, then is it simply—I say simply—an administrative delay, or something that is not working as efficiently and fluently as it should in the Department?

Ed Humpherson: We draw a very sharp distinction between—I am sorry to use this phrase again, but it is a good phrase—intelligent transparency cases where the numbers are not available, and cases where there is a risk of misleadingness: where there is a risk that the statement made misleads what the underlying data shows. In our experience, in cases such as those we are talking about now about the release of data, the statement is almost always supported by the underlying data.

There have been some other recent cases which have been more in the misleadingness space but, in fact, I would struggle to think of a case where, when the data are published, they clash with what the statement has said. I cannot think of a case. They are almost always backed up, which is what makes it so frustrating for us all because it is good operational information and it is relatively robust. It is good enough to support a statement, so why is it not being made available?

Q166       Ronnie Cowan: I am curious—I am asking for a friend here: if something is said in the Chamber that you think is the wrong information and you then ask for the data behind that to be published, is that the same process? Would they come to you? Can I come to you and ask, “Where is the data behind this?”

Ed Humpherson: Members of Parliament do raise cases with us exactly along those lines: what is the evidence underlying the claim being made?

Ronnie Cowan: You can expect an email then.

Ed Humpherson: We welcome it; this is our job. Thank you.

Q167       Ronnie Cowan: Given this situation, what escalation routes do you have?

Ed Humpherson: Imagine a ladder of escalation: the first rung is doing what I hope should resolve most cases, which is we see the issue—it is either brought to our attention or we spot it—and we raise it with the analytical leaders in the relevant Department. They acknowledge the issue, recognise that information should be made available, and make the changes. If that does not happen, we then raise it publicly by writing a public letter and making that available on our website to the relevant departmental leaders. If that does not work, our Chair, Sir Robert Chote, will raise it at a higher level, but if that does not work, I have mentioned perhaps drawing on your good offices as a Committee. We would be very happy to work with you in situations like that. That is essentially our ladder of escalation.

Q168       Jo Gideon: Over the course of our inquiry, the ONS has been criticised for revisions made to GDP, issues with employment figures and its handling of communications about the quality of gender identity data, and most recently its crime numbers. What has the OSR made of each of these issues? I am thinking specifically of issues around the recording of crimes that are more predominantly affecting women, such as domestic violence and sexual assault.

Ed Humpherson: In all those cases that you have mentioned, we have actively been engaged both before the issues emerged and then afterwards. If you look at our website, over the last five months you will see our judgments on the labour market statistics, on gender identity and on GDP revisions. So we are very actively engaged in ensuring that the standards of the code of practice are being complied with.

Before I talk about the specific cases, it is worth emphasising that the ONS is trying to do some difficult things. It is always hard to estimate GDP and to get the estimates in the hands of decision makers quickly, which is their intention. The economy is very complicated—there are multiple data sources—and, moreover, it is particularly complicated at a major inflection point. In the pandemic, there was a very significant exogenous shock: a shutdown and then a reopening. Those two things make it inherently hard to track what the economy is doing at those times.

It is similar with the labour market scenario: it is intrinsically hard to rely on a survey-based measure of unemployment when survey response rates are falling across the world. This is why the ONS is looking to transform its labour market statistics. On gender identity, this was the first time this question had been asked in a census; it was the first time the ONS had attempted to make an estimate of this part of the population.

So my first point is that the ONS is trying to do some difficult things. Having said that, it is our job to make sure that the ONS is complying with the code of practice. What ties those three scenarios together is not so much the question of the underlying statistics, but the question of how the ONS communicates and engages, how it explains what it is doing, how it responds to challenges, how it puts its outputs in the context of what is more broadly known. Across all those areas, the underlying common thread I would encourage the ONS to really think about is how it responds to users, to challenge, and how it communicates uncertainty.

Q169       Jo Gideon: Would it be fair to say that communication is a persistent problem for the ONS?

Ed Humpherson: It is an ongoing challenge and, as I say, it comes up very prominently in those areas I have highlighted.

Q170       Jo Gideon: What have you recommended the ONS does differently in the future?

Ed Humpherson: It is worth saying that the first conclusion of our report on GDP revisions was to say that its approach to revisions is sound and appropriate, meets international best practice, and that the most extreme criticisms it has received were misplaced. We recommended that it pay much more attention to how it communicates the inherent uncertainty of GDP measures, and it thinks about how that communication pitches into what is already known about the economy. If it is changing the economic story, if people are thinking the economy was in a particularly bad place and the revision changes that, that means that the weight of the communication needs to be greater.

The good news is we published our report in November. The ONS published its latest quarterly GDP estimates on 22 December, and you could really see it responding to our recommendation. It contextualised the numbers in a much better way and explained the different sources. On GDP, we also recommended thinking much more about alternative data sources and getting those data sources in more quickly because that might reduce the time it takes to establish a fuller picture of the economy.