Public Accounts Commission
Oral evidence: National Audit Office Audit Quality
Tuesday 23 November 2021
Ordered by the House of Commons to be published on 23 November 2021.
Members present: Mr Richard Bacon (Chair); Mr Nicholas Brown; Peter Grant; Dame Meg Hillier.
Questions 1-40
Witnesses
I: Gareth Davies, Comptroller and Auditor General, NAO; Elaine Lewis, Executive Director, NAO; Kate Mathers, Executive Director, NAO.
Written evidence from witnesses:
Witnesses: Gareth Davies, Kate Mathers and Elaine Lewis.
Q1 Chair: Welcome to the Public Accounts Commission. We meet to discuss the quality of the NAO’s financial audit work, in circumstances where we know that there is going to be change, because the Government have concluded that it is not appropriate for you, as Comptroller and Auditor General, to continue to be supervised by a body, the Financial Reporting Council, appointed by a Minister for the limited but important area of companies owned by the Government under the Companies Act legislation. We therefore know that changes are coming, and this is in a sense a shadow exercise, because we are moving towards what we expect will be a new regime where you will still appoint an organisation to review your quality but we, as a Commission, will then take their report rather than you. Can you set out, CAG, how you think that is going to work, how the approach will be beneficial, and what the key challenges are that you see ahead?
Gareth Davies: Of course. As you said, the key principle at stake here is the independence of my role and of the NAO as a whole. It is vital that any arrangement preserves that independence from the bodies that we audit, including all ministerial appointed bodies. It is very desirable, as you say, that we are not accountable to a Government body. As an officer of Parliament, I am already accountable to the Commission for the oversight of the NAO, its budget, and the delivery of its functions, so it seems appropriate for this Commission to hold me to account for the quality of the NAO’s audit work.
Obviously, as our day job is holding other people to account, this has the big advantage within the NAO of transparency, of emphasising the importance of audit quality in our work, and of demonstrating to all our staff how seriously that work is taken by Parliament. As you can see from the documents that we have already shared, audit quality is a top priority for the organisation. We are investing significantly in many aspects of it, which I am sure we will cover in the session this afternoon. For our staff to see that Parliament takes this so seriously that once a year, at least, it takes evidence from me directly, informed by the specialist inspectorates, which can provide you with high-quality feedback on our performance, sends exactly the right signal to our staff on the importance of that part of their work.
Q2 Chair: What criteria will you use when you are appointing an independent organisation to review the quality of your audits, because there is a potential—I am not saying that this is likely to happen, but you can see how some might regarding there as being a potential—for a case of marking your own homework, if you are not careful. How will you avoid that? What criteria will you use?
Gareth Davies: We already compare ourselves with the top firms in the audit market in the country, and in fact in the world. That seems to be the only relevant standard, because we audit hugely complex and large organisations and it is right that, when we ask somebody to inspect the quality of our work, they are people who are equipped to assess those kinds of audits. That is why the voluntary arrangement that we have had with the FRC in the last few years has been in place, because the FRC, as you know, inspect the biggest firms. They inspect audits of a similar size and complexity, albeit in the private sector rather than the public sector. That is why we have worked with the FRC in that capacity to date. There has been no suggestion at all that they have held back in their feedback to us or given us an easy ride in comparison with anyone else. On the contrary, I think their reports are approached in exactly the same way as they do for any audit firm.
Q3 Chair: Are there any deficiencies in the Financial Reporting Council’s approach that you would like to see remedied or improved, or aspects of the way that they work that you would like to see retained in the new system?
Gareth Davies: The reviews that they carry out are a high-quality assessment—sometimes painfully high-quality, from our point of view. We think that they are a very good dipstick into the quality of our work. We have an annual conversation with the FRC about whether they can bring their work forward. That is the main point of discussion with them at the moment, because obviously the earlier that we get the feedback on a particular year’s audits the sooner we can make improvements for the audits to come. At the moment, the difficulty that we have is that essentially two cycles go past before you see the impact of improvements that we have made.
Q4 Chair: Yes. Is there any scope for doing what you internally call hot reviews? During an audit, you dipstick test the quality and make possible changes during that process. Could an external scrutineer of your work be doing the same?
Gareth Davies: Obviously, as you said, we do that already. I think it would be very difficult to co-ordinate the timing because audits move at a reasonably high speed, and it would be very difficult to bring an external party in for hot reviews. We do them ourselves, and obviously we do make changes and improvements based on them, but the value that the FRC bring is that they are inspecting all the firms at the same time as us. We benefit from their feedback about what the leading practice is in the rest of the sector, which we obviously cannot generate entirely ourselves. That is why the sooner we get their feedback, the better for us.
Q5 Peter Grant: Good afternoon to all our witnesses. Mr Davies, in reply to the Chair you mentioned that you are benchmarking your own performance against the top audit providers in the world. Twice in my lifetime the entire world financial system has come close to collapse because all the top audit companies in the world got it wrong at the same time, in the same way. I know that in some of your own NAO reports you have referred to the danger of group-think among Government Departments. How do we guard against group-think among all the top auditors? How can we be assured that everybody is not missing the same potential risk, as they have in the past?
Gareth Davies: I was making the comparison to illustrate that we hold ourselves to the highest possible standard. I was not saying that we regard the firms as the benchmark for perfect quality, because as you can see from the public reports, they have similar feedback to ourselves, with room for improvement. So you are right: we certainly do not regard that there is a level of perfection out there. But the way to drive professional improvement is to challenge ourselves based on what others are achieving, and where others are moving ahead faster than we are, it is important to know that.
On your question about how we avoid group-think, the FRC has the key role. Its job is to challenge us and all the firms on the quality of the audits that are being delivered. I think it has done a good job in recent years, actually, in prompting quite rapid change in audit practice. Audits now are pretty much unrecognisable from 10 years ago. A lot of those lessons were driven by the financial crisis. Some of the areas that I am sure we will cover today relate to the audit of complex financial instruments, which was at the heart of the problem in 2008-09. Audit practice, prompted by the regulators, is fundamentally different now in those areas.
There are examples where the system has driven change and improvement. Whether that is fast enough is a matter of opinion, but it certainly keeps us on our toes. People feel prompted to challenge the way we do things, and it is certainly regarded as a tough regulatory process when you are on the receiving end.
Q6 Peter Grant: How do you assess the similarities and differences between the assurance process that you go through and the process that applies to your counterparts in the devolved Administrations? Are there lessons that we could learn from them and lessons that they could learn from us?
Gareth Davies: We have regular liaison meetings—I have them with my counterparts in the devolved Administrations—and one of the strands of our co-operation is on audit quality, so we compare approaches. We have a much more developed approach in some ways because we are the only one of the four to audit Companies Act audits, so we are the only one of the four that is inspected by the FRC in the way we have been discussing.
Clearly, that has driven a level of attention to those areas in our case that has not yet applied across the other four. They have similar arrangements, in that they have commissioned external bodies—in their case it is usually the institutes, so either the ICAEW in England and Wales or ICAS in Scotland—to carry out independent inspections of their audits in the same way that the FRC reviews ours. They are publishing their results in a very similar way to us. We challenge each other on what we are doing to improve. Of course, the fact that we are auditing big and complex UK-wide institutions does not mean that we cannot learn from innovations in Scotland, for example, as we have done. That channel of communication is live and working quite well.
Q7 Peter Grant: I am always in favour of other people adopting good innovations from Scotland. Finally from me for now, looking further afield, what international comparators would you look at and say, “That is where we like to be. That is what we could learn from.”?
Gareth Davies: There is a very diverse field when we consider our equivalents in other countries. Many audit offices are not accountancy-based bodies and do not carry out financial audits. They rely entirely on subcontracting all that work to the private sector. Quite a few audit offices in Europe, for example, are legal institutions staffed by lawyers rather than accountants, so there is actually a surprisingly small comparator base for organisations of our kind that carry out financial audits.
We make the same kind of links on matters that I was describing in the UK. It is harder to learn direct lessons, to be honest, because the arrangements in place around the world are so diverse. Not all, for example, use IFRS-based accounts. Not all would recognise the same basis for auditing as the one that we adopt.
The US is always a good comparator for us. The GAO in the US does carry out financial audits, so we stay in touch with them on the approach that they take. But again, the detail is different there. It is very different on the private sector in the US with Sarbanes–Oxley and the very different control framework, but we still have fruitful exchanges with the US.
Q8 Mr Brown: Nevertheless, what were the key points that you took away from the FRC’s latest audit quality report? I listened carefully to your last answer and, to some extent, it is reasonable.
Gareth Davies: First of all, we agree with the findings, and we are already acting on their recommendations. One key area that they highlight, where they express a bit of frustration in how long it is taking us to address their point, is the audit of complex financial instruments. Of course, there are more of these around post financial crisis, and even now post Covid, than before. This boils down to rapid developments in accounting and audit practice on things like credit loss estimates built into those financial instruments. Essentially, we are being challenged on how deeply we go into challenging the departmental estimates, the models they use to make those estimates, and the data that goes into the models. That is now a very sophisticated process, both on the account preparer side and the audit side, for getting into the detail. Maybe I will ask Kate to explain how we would approach the audit of a complex financial instrument, just to illustrate some of the changes we have made in response to this feedback.
Kate Mathers: Just to explain the approach we take to trying to build in quality, the heart of this is trying to understand the risks that sit within any given set of financial statements. If we are considering financial instruments, we will have an initial consultation meeting with the team, which will be chaired by myself or my colleague, Elaine Lewis, as heads of audit, to work with the team to help get our initial assessment in the right place. Where we identify that there may be some complexity around a financial instrument or another area, we will then consider the approaches that we would take to audit that.
You work closely with the team to really go beyond the surface level assessment to get into a more granular understanding of what we are actually grappling with here. As Gareth was saying, that includes really thinking about the assumptions that might be built into the model—the data input, the integrity of the model itself, and how that works in terms of calculating the values for the financial instruments. That is the initial step, and it really sets in train a whole process to make sure we are thinking about quality at every point in the audit. The risk assessment is the first part of that. Following on from that is the execution of the audit approach and the design of the audit approach.
If we are thinking specifically about financial instruments, as Gareth said, there is a degree of complexity to them. We have some in-house experts—corporate finance experts—which we are able to draw on, and we have been building our centre of expertise in financial instruments for a year or so now. We have made some investments there, dedicating some senior time to that and also working with external experts here. We recognise that we need to grow our in-house knowledge and expertise, and we need to draw in external support and expertise to look at some of these complex instruments.
We typically work with some of the firms who perhaps have more experience in these areas. We have contracts in place with a range of firms for specialist expert audit support, which we draw on to help bolster our own expertise when we need it.
Q9 Mr Brown: How do you satisfy yourself that you have really got under the skin of the complex arrangements that you are exploring? How will you measure progress within your own organisation that you are getting on top of this complex and difficult area, and that year on year you are the master of it, rather than running behind the problem?
Kate Mathers: The ultimate indicator for us will be the scores from the AQR review, so we want to see those improving. We want to see the level of comments in this area diminishing. That is the ultimate key performance indicator for us, but we also work through our own internal quality indicators as well, so we might assess the level of consultation with teams, how quickly we are surfacing issues and how promptly we are able to respond to them. An important quality indicator is to surface the issues early on in the audit process, so that teams have adequate time to really think through the issues and draw in additional support or expertise where they need to do that. There is a range of things that we can do to monitor our own quality—the external review, the in-flight reviews, as well as the external benchmark reviews.
Q10 Mr Brown: But what will you be able to show the Committee next year that shows us that you have actually moved things forward? I am looking for an objective indicator.
Gareth Davies: Fewer prompts from the FRC in their report, which we will share with you, on this topic. In fact, what we really want are positive comments about the progress we have made from the independent inspectorate, so that is what we will be aiming for. That is your best independent evidence that we are making progress on this.
I would stress that although we completely agree with the need to make the improvements that we are talking about, because we are talking about some of the financial assets that Government had acquired in the last 10 years, obviously we also audit the transactions that Government have entered into in the last 10 years to dispose of some of those assets when they have been marketable, and all the evidence so far is showing that the valuations that we had audited in the previous years have held up through the sale process, and that value has been realised for the taxpayer. That is a useful bit of assurance that those were based on real values.
Q11 Mr Brown: You have set yourself the target of 100% rather than the industry average, as I understand it, of around 71% now. You are going to march remorselessly in that direction. As you make progress, and I have confidence that you will, how much effort will you have to put in to capture the last bit of that ambition? It is not the wrong ambition to have, and I am not critical of it, but it seems to me that striving for the last bit of that will be quite difficult and may involve disproportionate—
Gareth Davies: I think the honest answer is that this is an endless journey; this will not stop. Because, as we have seen in the last 10 years, auditing standards develop in response to changes in financial reporting standards, and if we ever come to you and say, “We have successfully achieved audit quality and we are going to reduce our investment in that now,” that will be the time to worry. We are going to have to stay on top of developments, continually update our staff’s expertise, and challenge ourselves that by focusing on financial instruments, for example, as a big priority area, which it is, we haven’t taken our eye off another ball somewhere on our audit quality. I don’t think I will ever be coming here to say we have cracked this and you can stop worrying about it; I think it will be permanently in our list of priorities and we will be running to stay in the right position.
Q12 Chair: Can I bring in Elaine Lewis at this point, because I think I am right in saying that you are new to the job? It is fair to describe Kate Mathers as head of audit and you as head of quality; I think that is right. I am sure you will be delighted to know that the Comptroller and Auditor General thinks there is a need for more investment in audit quality, but plainly this is a relatively new area for the NAO. The issue of financial audit quality has been the highest thing on the risk register for the NAO for the last three years. Can you speak to your plans to do something about that, as you have taken on this new role?
Elaine Lewis: Yes. Clearly, my appointment was not the first time that we had focused on audit quality, but I think it shows an increased focus by senior leadership on this, and also bringing together lots of work that we were already doing. We have set out the key developments in our audit quality plan, and I think the way to characterise that is as a whole-systems approach, so it is making sure that the whole of the NAO is set up to support teams in delivering audit quality. So it is not just about individual teams; it is about whether you have got the right resourcing model, the right training, the right support and, really importantly, the right culture that encourages people to focus on audit quality, to consult on audit quality.
Q13 Chair: How do you incentivise the process to place an emphasis on quality? The FRC referred to this. It is one thing to say that you need to have more emphasis on culture—everyone talks about culture. How do you create the architecture internally so that the team thinks more highly of the importance of challenge, of the right kind of culture, in the way that the FRC has been talking about?
Elaine Lewis: I think the focus on quality, actually, for our people has always been absolutely key. People want to do high-quality audit work, and I do not think that has particularly been a challenge; it is how you deliver that. You are right: the difficulty—this is something that the whole profession is dealing with—is how do you have that culture of sufficient challenge of management, as opposed to corroborating what they are saying? I would say it is a bit of talking about it, but it is more about rewarding those behaviours when they take place, and showing that that is the kind of thing that we want to see. For example, we have had occasions when we have delayed the signing of an audit certificate because we have felt we have more work to do, showing that audit quality and sufficient challenge of management takes precedence over achieving a timetable. That is not to say that timetables are not important, but just really showing the primacy of quality there.
Q14 Chair: At the same time, you will have those managers going to see the CAG who is pressing them to get the thing delivered on time. There is an inherent tension there between the two.
Gareth Davies: I always insist on the job being properly done. I will back up our teams who might be facing a difficult conversation with the audit committee of a Department about the delivery dates. If we need to explain that more work is required, I will always back them up to do that. On your question about how we incentivise in our systems, every auditor from the top to the most junior member of the team has a quality objective in their performance objectives, all of their performance reviews focus on audit quality, and we check that that is happening in a systematic way.
Q15 Chair: How do you measure it?
Gareth Davies: The feedback from our cold review, our own internal reviews and any external reviews that have affected their audits is discussed openly and shared. People know that we are going to do that as well, so they can see how seriously it is taken. We have a balanced scorecard for running the NAO, which has four quadrants, with people, delivery, quality and finances all adding up to our overall impact. Every month, every team knows we are tracking quality in that balanced scorecard. You cannot be in the NAO and not realise how important it is and how much we focus on it.
This is part of our strategy review. When I arrived we refreshed the NAO’s values. It is easy to be cynical about organisations and their value statements, but it was a bottom-up request from the staff that they did not think our values then fully reflected what was important at the NAO. So we have had four new ones for the last two years. One of them is excellence, and that does set a tone. People can see that we are really serious about being the best at what we do here.
Chair: Now we have Meg Hillier and Nick Brown both eager to get in.
Q16 Dame Meg Hillier: As CAG you have brought in people from outside at senior level. Your predecessor did a bit of this. How important is it to bring outsiders in to help you maintain the standards and learn from the other bits of that sector? Do you have a plan to do that at lower levels of the organisation?
Gareth Davies: We have successfully done that at every level. At the very top, on audit quality we have formed a new audit quality board chaired by one of our non-executives, Janet Eilbeck, who has a public audit background and a distinguished career in PwC working on public audit, and another non-exec, also a former audit firm partner. For the last year or so they have been holding us to account for the progress we have been making against our quality plan. That brings a really useful external perspective.
Down at team level, we have appointed directors with private sector as well as public sector experience. When we took on the audit of the BBC, we bolstered our audit director grouping with some people with commercial experience, and they have stayed and have made a big impact on the work that we have been doing and the way the teams look at the different elements of the job.
Obviously, we recruit from all over as we go down through the skill mix. It is really important for us as an organisation focused on one particular sector to draw in expertise from across the board, and that is working well.
Q17 Mr Brown: The relationship with the Bank of England is relatively new. Because of the very nature of it—two strong institutions, highly respected in our country—there are slightly conflicting aims. How, for example, do you audit quantitative easing and provide some reassurance to the Government, the taxpayer and the ordinary citizen?
Gareth Davies: The financial audit arrangements for the Bank of England are different from the value for money audit arrangements. We have value for money audit powers for the Bank of England. We are not the auditor of the accounts of the Bank itself. That is still one of the firms. We audit a couple of the subsidiaries, particularly the ones engaged in the covid lending response. I think it is still EY, if I’ve got that right. It certainly was EY on the audit of the Bank of England. It is now KPMG, and they have just rotated. We obviously work with them as auditor of some of the subsidiaries, so we understand the issues that are arising in their audit, but we do not actually sign the audit opinion on the Bank itself.
Q18 Mr Brown: How much reassurance should we take from that? Let me ask that another way: is that the correct relationship, in your view?
Gareth Davies: At the moment, it is working well, in our view. It means that we have been able to focus on what, for the Bank, was a very unfamiliar process of being held to account by Parliament based on reports from the NAO. We have looked at quite a few important areas. We started with what you might regard as back-office areas, such as office accommodation and IT and so on. Recently, we have gone into more operational territory, such as the production of cash and the impact of the reduced use of cash on their plans. I think that has been a successful process so far and is delivering what it was intended to do, which is the Bank’s greater accountability to Parliament.
Q19 Dame Meg Hillier: It is interesting looking at what the FRC has done; given the hundreds of audits that the NAO do, they have looked at seven. Do you think that is a big enough sample to draw any generalisations from? Would you be up for having some of your audits being opened for a deeper dive?
Gareth Davies: The audits that are inspected do get very deep dives. They are already deep; the question is whether there should be more of them. This is a regular conversation between the FRC and us. I think our portfolio is growing, so naturally the number will go up with that growth. It is the FRC that determines the sample size that is necessary on our Companies Act work, because they are in charge of those decisions. The voluntary sample of our Government audit work that we agree with the FRC is more of a negotiation between us.
Obviously, we want an accurate view; the issue is both the FRC’s resources and our resources—frankly. It is a live discussion. That number is only going one way, and that is upwards. The question is, how quickly? These are resource intensive, both for them and for us, so we have to be careful about that. However, I would expect that number to grow in the years coming up.
On your other question: I do not think we can ever say it is a statistically representative sample. I do not think the FRC would claim that for any of their audits of firms, either. However, it is an extremely useful one. The reason that we take it so seriously is that it is a very similar picture to the one painted by our internal review. It is not as if we get a very different story from the larger number of internal reviews that we do—we get a very similar picture to the FRC. In that sense, I do not feel like we are somehow missing something or getting a distorted view from their sample.
Q20 Dame Meg Hillier: The samples are going to be relatively small from your Companies Act work, because you do not have many of them.
Gareth Davies: Yes.
Q21 Dame Meg Hillier: What are the barriers, if there are any apart from resourcing, to having more work done on your other audit work? There is an awful lot more of those to choose from.
Gareth Davies: It goes back to the point I made right at the start, which is the timeliness of the FRC’s reviews matters to us as well. I would be worried if a larger number meant that they were not able to bring that work forward, and so we were still getting it very late in our cycle. Coming up with an amount that gives us good assurance that we are not missing any important developments but is a good use of both the FRC’s and our resources—that is the sweet spot we need to hit on that.
Q22 Dame Meg Hillier: Do you have any idea of a number? You say that it is going up—or that it will go up—but do you have any idea of what to, and by when?
Gareth Davies: Elaine might be able to help on this, but I think they are planning to increase the Companies Act sample from four to five in the year they are about to review. We are still talking to them about the number on the Government work, which is currently three but may go up to four. These are small increases, but obviously in percentage terms that gives us nine rather than seven as a total sample.
Q23 Chair: It is still quite small when compared to what the firms do when assessing their own audit quality work, is it not?
Gareth Davies: We do that as well. This conversation is just about the sample picked by the FRC. We have a bigger sample; Elaine can give you some idea on the size of our internal samples.
Elaine Lewis: I think we did 20 last year.
Q24 Chair: That is much more in line with the firm’s benchmarks. Elaine Lewis, I will return to the subject of your role as head of quality. Does the NAO’s own internal review tell a different story from the FRC, or is it broadly the same?
Elaine Lewis: No, it is broadly very similar. The scores are pretty much on a par, round about 70% again.
Q25 Chair: What are you doing to improve the root cause analysis of why you are not meeting quality standards? Because it is not a new thing. I was reflecting a moment ago on the CAG’s appointment of you. You have been in post now for how long?
Elaine Lewis: Two years.
Q26 Chair: I remember that the run on Northern Rock was August 2007, and most of the financial crash happened from January 2008 onwards. So, we are talking 12 years ago. As an institution, you have now decided to invest in a much more substantial way in quality, and have a head of quality, in a way that you did not. It might be fair to say that the response has not been electrifying, but you are getting there. When are we going to see the sorts of improvements that we think we ought to? Is there a risk that the NAO has been a little complacent?
Elaine Lewis: In terms of the root cause analysis that you picked up at the start there, the FRC reviewed our root cause analysis work as part of their review last year. They found a number of areas of good practice in that. Having said that, there were some areas for further development. That is something where we are always learning and looking to develop further.
I think we have a relatively good story to tell on root cause analysis. The key, as you say, is how we learn the lessons from that, not just for that particular audit but across the practice as a whole. That is what we are striving to do. We certainly recognise that this is not something you can do as a one-off; it is taking a number of years. We think the focus is starting to come through; we are starting to see some of that, but it will take time.
Gareth Davies: May I add a reflection on that? Coming from a firm that had the same quality improvement agenda, and comparing with the NAO, there are some common features. Overall, the big change we needed was to be much more open internally with the whole organisation on audit quality. There was a slight sense of fear around this, for some good reason, recognising it is important to get it right.
Rather than that spurring lots of exchange of experience and ideas, and information about how to improve more quickly, people were clamming up and not having those exchanges. The big shift in that time is, first, in a very structured way to have those open conversations. Some of the braver directors, who maybe have had quite bruising reviews, actually describing exactly what had happened, why the findings were the way they were, what they are now doing differently on their audits—that was not happening before and it is happening routinely now.
That has made a big difference to people’s understanding of the challenge, and that this is not some difficult mystery that, if only you had the key to it, you would be able to do perfect audits. It is about hard work, learning, application of that learning, and then repeating that cycle. That is already making a difference. What was holding back the speed of improvement—which is behind your question—that is the core issue. It has significantly improved. I do not think we can say we have completely cracked it, but it is moving in the right direction.
Q27 Chair: I want to bring in Peter Grant in a moment to talk about the audit of complex financial services. You have referred to that several times. I would like to clarify something. Plainly, some of this added complexity is due to the assets that the Government ended up owning following the financial crash—these hideously complicated toxic assets. Mr and Mrs Taxpayer bought them, whether we liked it or not. Now you are trying to do something about it and the first thing is to understand that.
That is plainly part of it, but isn’t it also true that the nature of the changing world of international financial standards reporting is more generally applicable? Government owns a very wide range of assets, whether it is the NHS, the property estate, which is many tens of billions—and the same for the Ministry of Defence—and many other areas. It invests in many other kinds of things, such as software that is good to start with and then has an asset value that depreciates and becomes legacy and needs replacing.
One could repeat that in many different domains within Government. They all have risks and complexity and are all subject to changing international financial reporting standards, which eventually affect everything. To what extent is this a problem because of those toxic assets from the financial crash, and to what extent is it a much more general problem that will make it difficult for you to reach that threshold, which Nick Brown was talking about, of getting it perfect every time?
Gareth Davies: Perhaps I will kick off and Kate or Elaine might want to add. I think it is both. As you say, the Government’s activity in these areas has been very different over those phases. The covid experience, again, has been another big impact, with Departments that have never had to administer a loan book suddenly finding themselves with loans to, say, cultural institutions in the Department for Digital, Culture, Media and Sport, and so on. So there is now another wave of new financial reporting risks and audit risks arising from the pandemic, and that process continues.
But there has also been, as you say, partly in response to those events, a significant change in financial reporting standards and audit standards. In some ways, what I meant by this quest for higher quality needing to be a permanent one is that that is the nature of a professional audit. The world will keep changing around you. That will present different and changing risks and your job is to be on top of those and produce audits that cope with those changes successfully. First, we clearly need to demonstrate improvement from where we are, but then we need to keep it up and that will take continuous effort. Kate, do you want to add to that?
Kate Mathers: One of the programmes that have happened already is our audit transformation programme, which is an investment that we are making to transform quite fundamentally how we are doing audit at the NAO. That is to respond to exactly those kinds of challenges we now face and make sure we can keep aligned with or, indeed, ahead of those. We are making substantial investment there to change our methodology so that it is much more focused on a much deeper understanding of risk in these complex areas in a way that helps guide our auditors through what they need to look at there.
You are quite right that there is a whole range of complex balance sheet assets and liabilities, and it is not just from covid. We have to think of the financial support measures that the Government have put in place as a result of the pandemic, but other things as well. The student loan book is a good example of the complexity there. There are the provisions for decommissioning nuclear waste. They all have high levels of inherent risk and great degrees of judgment in there. Those are the assumptions and aspects that we need to work through very carefully when we are conducting the various audits of different Government Departments.
Q28 Dame Meg Hillier: On that point, on the Public Accounts Committee, with your help, we have grappled with some of the long-range issues, such as nuclear decommissioning. We know that there are very few financially qualified permanent secretaries and we increasingly have seen some of the advice notes that they have had. Their accounting officer assessments are drafted by more expert financial professionals in Departments. Do you think Departments have a good grasp of this? I am sure your work helps them, but do you think there is a deficit in Whitehall in understanding some of these complex measures, which often have been landed on Whitehall, as the Chair highlighted?
Kate Mathers: I think it is certainly right that Departments have had to respond at pace, certainly over the last 20 months or so. We can see that the Government financial profession is growing and developing, and we work closely alongside them to tackle some of the very complex issues that Departments have to grapple with to bring them to account and that we have to audit with them. In the same way that we are growing our expertise, that is certainly the same challenge for Departments, and we can see ongoing improvements in that area.
Gareth Davies: I think the role of the Government Finance Profession is crucial there because that is how Departments that may be struggling to access expertise and capacity efficiently. We have seen good examples of that working in practice. As Kate says, we attend the meetings of the finance leadership group around Government so that we can talk about exactly these issues and say, “This is what the auditor is going to need to be able to see to sign off your increasingly complex accounts.” So we just need to keep going on that exchange of views so they can invest themselves.
Q29 Peter Grant: On the audit of either organisations that are entirely involved in financial services or Government Departments or Government-owned companies, where complex financial instruments are a significant element of the balance sheet, the Financial Reporting Council report seems to have identified an unacceptable trend of poor audit quality in those higher risk and more complex audits. Were you surprised at the strength of the criticism in the FRC report?
Gareth Davies: I was not surprised that they mentioned it. I would have worded it slightly differently, but that is what permanent secretaries say to me about NAO reports all the time. Sometimes you have to suck it up. We do not disagree at all with the thrust of their point here, although there are examples in the last year of audits in their sample with financial instruments—the Low Carbon company is a good example—where they regarded the audit as satisfactory. What they are saying is that, too many times, it is those kinds of audits where they find weaknesses in the approach that we have taken in challenging the detail of the financial models. That is why we have set out the additional investment. We created the centre of expertise on financial instruments a year ago. We are putting more money into that and working more closely with experts from outside the NAO on that issue.
Kate has given you a good description of the process we go through in doing a risk assessment and then identifying the specialist resources that we need to plug into the audit to deal with the risks that were identified. We have a good modelling team at the NAO, which is pressed into service so often now that we have had to increase its size again this year. That helps us look at the models used by Departments, pick them apart and identify any flaws in them. We are getting to the stage whereby we need to create our own test models to compare our results with the ones produced by the models of the Departments. That is increasingly standard practice for things such as banking audits. We need that capability ourselves, so we are also building that up. I think we have explained how seriously we have taken that point, because the life of some of these assets on the Government books is so long that this will be around for some time, so we need to respond to the feedback.
Q30 Peter Grant: These are also the kinds of elements on a balance sheet that are intrinsically much more at risk of overstatement or understatement. If it is a hospital, you can tell it is a hospital. You cannot put a non-existent hospital on a balance sheet, but you could put financial instruments on a balance sheet that are not actually worth anything. Given the higher risk associated with that part of a company’s financial statements, why is it that, six years after the FRC started raising concerns, they are still having to raise it and have now gone as far as to say it is unacceptable? What has happened in the last six years that has not achieved what it should have done?
Gareth Davies: I can talk about the last two. This has been a big investment priority for that time, including the setting up of the centre of expertise that we have described. I can certainly demonstrate the progress we have made in that time, but I do not know whether my colleagues have any insights from further back.
Kate Mathers: I think we have recognised the challenge there, but what is really clear is that the standard—the quality mark—has moved faster than we have moved over those years. The substantial investment that Gareth was talking about should help us pick up the pace and move forward quicker. Clearly, there is more for us to do in that area, and that is a critical area of focus for us.
Q31 Peter Grant: I hope you will not mind my saying that “I wasn’t here two years ago” is a response that those of us on the Public Accounts Committee are quite used to hearing from permanent secretaries. May I ask the follow-up question that I sometimes ask permanent secretaries? When you came into post and had your handover from your predecessor, were you aware at the time that this was one of the issues that you had to get to grips with immediately, or was it only when an FRC report landed on your desk that you realised it was an issue?
Gareth Davies: No, because obviously part of my due diligence before taking up the post was to read around everything to do with the NAO, including the available information in the transparency report. It is certainly not right to get the impression that work on this started only two years ago; it has been a long process.
I think the key point is the one that I made earlier: the big impact that I have made is to get the organisation to open up on this, internally and externally. We now publish our AQR scores on our website in a way that was not happening before two years ago. I have explained the internal conversations that are happening, which were not happening before, and that is the key. There is a lot of good work going on, but it was not being shared efficiently and was not having the traction that it should have had. Combined with the specialist investment that we have described on these financial institutions points, that is taking us faster to where we need to be.
Gareth Davies: Yes, in the sense that we are increasing the size of the centre of expertise beyond what we felt was originally needed, so that is definitely a change in our plans. This is not all about permanent appointments. We need to just buy in some of the skills we need when we need that particular input on a particular audit. The framework contract that Kate mentioned for specialist expertise, which we have just re-tendered with this in mind, means that we now have a good range of potential suppliers with this kind of expertise to call on. It is important that we have a good choice. Quite often, we will find conflicts of interest in particular areas. We will find that a particular firm has advised the Department already on the estimate, so we can’t use them as the auditor’s expert in that case. We have a good number of firms on there, which is proving effective.
Q33 Peter Grant: Another concern that the FRC raised related to judgments on materiality, which is clearly fundamental to the way that an audit is carried out and fundamental to the way that the findings are reported. The question is probably to Kate Mathers rather than to yourself, Gareth. How do you make sure that the staff undertaking the audits have sufficient expertise to be able to judge where to set the level of materiality? What support do they have in that work?
Kate Mathers: Absolutely, it is a key part of any audit. It drives the amount of work that you need to do. It helps you form your judgment. It sits behind your professional judgment.
We take the point that the FRC has raised with us. We have issued more guidance to our teams, in terms of training and support. We cover that at our planning and consultation meetings. We have a specific deep dive into that with teams from some of the high-risk areas. We are making the expectations much clearer in the guidance and within our methodology as well.
The FRC puts together really helpful thematic reviews on what you see happening across the wider profession here and has put guidance out on materiality and the benchmarks that you might use when setting your own materiality and planning your audit. We draw quite extensively on that, in order to benchmark our approach to materiality as well. So the answer to your question is training and support packages for our teams and regular reinforcement and technical support to help ensure that teams are in the right place with this.
Gareth Davies: I think the feedback is not so much that the FRC thinks we have chosen the wrong level of materiality. It is more the quality of the evidence on file to explain the choice we have made. That is a common issue, actually—the quality of the documentation of the judgment made by the auditor, rather than the judgment itself. We have had a very specific challenge on, “It wasn’t clear from the file how point X had been taken into account in setting materiality from this organisation”—they would have expected to see more on that. That is the nature of the feedback.
A lot of our sharing of good practice is sharing examples where materiality decisions have been documented successfully and well, so that picking up the file as a reviewer, you can immediately see how those judgments have been made. That is the standard we need to get to everywhere.
Q34 Peter Grant: Can we look at the importance of materiality during the audit? It is clearly a big factor in deciding how much testing needs to be done. At what level in the organisation is the decision taken as to how material something is? And, related to that point, what are the red flags that the auditor on the ground would be expected to recognise and take back to someone more senior and say, “This is bigger than we thought”?
Kate Mathers: You set materiality out at the start of your audit as part of your planning process. That will drive the size of samples and the work that you are doing there. That is a quantitative basis on which to set your materiality. Of course, once you get into the audit, you need to consider qualitative factors as well. Something could be very sensitive and a matter of significant public interest. It is not simply driven by the numbers, if you like.
We also give very careful consideration to the issue at hand as well, and that will also drive the amount of work. It also drives what we report on at the conclusion of the audit. If there are material concerns and we think there is material fraud or material error, Gareth will report on that and qualify his audit opinions. The judgments are set by the team at the outset, but where we have significant concerns that will be consulted on at senior levels in the organisation, right up to the CAG.
Gareth Davies: Specifically, every one of our audits is the responsibility of a director. It is essentially the engagement partner, in firm speak. The director is responsible for signing off the materiality level at the planning stage in the way that Kate mentioned, and then escalating any conversations about materiality to either Kate or Elaine, and then to me as necessary.
It is worth saying that we give two opinions on Government Departments. There is the true and fair opinion on the accounts—the same as a company would have. We also give a regularity opinion on whether the resources have been spent in line with the wishes of Parliament, and we apply materiality to that in a different way. That is much more about the level where we would expect Parliament to be interested in any departures from the normal spend, so quite small amounts can be material on regularity in a way that they would not be on the true and fair opinion.
Our teams are very used to handling both types of materiality, and they involve me immediately if there is a question of qualifying our regularity opinion, as we are doing on an unusually large number of Government Departments this year in the light of the covid spending.
Q35 Chair: While we are on things like regularity, perhaps we can mention journal testing, because the FRC found that the audit team did not perform appropriate procedures over aspects of journal entry testing, including not sufficiently testing the completeness of manual journal entries. Of course, there is something worse than irregularity, which is impropriety. I am just wondering how you react to the fact that the FRC identified this. How concerned are you by it, because there are ways that management can override the audit systems if journal testing is not adequate?
Gareth Davies: This is one of the more frustrating ones, because it is a bit of an old chestnut. We should not still be seeing these kinds of comments in the feedback to us, because this is a well-known issue. Without going into too much detail, the first challenge is correctly identifying the whole population of journals, so making sure that we have a complete list. A journal is the way in which an accountant will achieve an adjustment to the accounts that is not based on a real-world transaction, so it is not generated by an invoice or a payroll payment. It is usually for technical adjustments to things like valuations and so on. As you say, it is very important because it would be an easy way to manipulate the accounts if you were so minded. The audit process must identify all the journals that have been applied and then, once we understand the population, search through that for any signs of irregular journals.
Q36 Chair: Do data and analytics mean that we are basically past that now—that you are looking at the entire population rather than a sample?
Gareth Davies: Increasingly we are. You have to make sure that you have the whole population for that to be an effective procedure. We are then looking for features such as journals involving senior people who would not normally be involved in those kinds of routine transactions, or any sign that controls are being overridden in the system or that journals are happening at an unusual time of the day, and so on. Those are increasingly sophisticated tests that we apply using data analytics. It is frustrating that that is still appearing in the feedback, because the answer to this is not that there is some new auditing standard, or a new complex arrangement out there. This is something that we must be getting right on a regular basis.
Q37 Chair: How do you think the quality of your audit work compares with the private sector, and how do you measure it?
Gareth Davies: The simple answer is that it compares pretty well at the moment, if you use the AQR report. In fact, they say that we are essentially performing at the level of the top firms. That is not good enough, though, because they have criticisms of the firms as well. That is the comparative measure, but our absolute measure is to be better than that. I think Parliament should take some comfort from our scores. On this inspection regime, which is designed for some of the most complex audits across any sector, our audits perform to the same level as the top firms. Below that headline measure, I think there are areas where we have more expertise, because we specialise in Government auditing. I am talking about regularity, for example. I think we lead the way on that, because it is not a concept that the firms will be using.
Q38 Chair: Do you think the fact that the NAO has such a key role in accountability for public expenditure means that your audit quality should be higher?
Gareth Davies: Yes. That is why our ambition is to be 100% on all the inspections. I think we should be the best auditor. We face a different set of risks to the audit firms. It is not easy to generalise too much, but we do not have the commercial pressures that sometimes cause problems with the firms. My auditors are not out trying to sell work to other people. They don’t have to argue with the consultants in the firm, who are cross-selling to their audit clients. We do not have those threats to our independence. We are under time pressure, as are the firms, so we have that one in common. Very few of the firms have the same year-end for all their clients, so that bunching effect of all of our work peaking at the same time is a real pressure on the NAO. It is one of the reasons we work with outside expertise to share the load.
If you asked me what the biggest threat is to quality, apart from all the cultural issues we have talked about, that sheer timetable pressure of the summer recess—which is, I think, the right time for Parliament to have audited accounts by—is a huge peak in our workload. In the past sometimes the need to deliver by that peak has had too much sway over some of these audit judgments. As we were describing earlier, we have been really clear with teams that those deadlines are important, but they must not take the place of properly evidenced audit work. If necessary, we will approve a delay if that means a better-quality audit as a result.
Q39 Chair: This is probably one for Kate Mathers: how do you keep pace with developments in the audit profession and the private sector?
Kate Mathers: There are number of ways. First and foremost, we take account of the FRC’s findings, both the specific findings that they raise on our audit work and the broader thematic points that they raise, so we have good insight into what our regulator considers to be good practice. We also have relationships with the firms through our frameworks, which we have talked about. The firms carry out a number of audits for us; a proportion of our work is contracted out. We work in partnership with the firms. The idea is that, as well as carrying out the work, there is some exchange of practice and knowledge. We also buy in the most specific specialist expertise where we need to supplement what we have in-house. So there are a number of ways that we keep across what is happening in the wider profession.
Chair: Nick Brown, I think you had a question about the pandemic.
Q40 Mr Brown: The FRC was very complimentary about the organisation’s response to the covid pandemic. I know that it may be too early to ask the question, but is there anything you did that you think you would retain and that would strengthen the long-term working of the organisation? To ask the same question the other way around, is there anything you did that, in hindsight—that is a marvellous tool for decision making—you wish you hadn’t done or that is a lesson to be learned?
Gareth Davies: Yes to both, actually. If you had said to me a month before the thing kicked off in March 2020 that we would complete that year’s audits and do the entire set of next year’s audits working remotely, I would have said that we were not prepared for that and that it would be a very difficult proposition. It was difficult, but the fact that we were set up to work remotely anyway as auditors working on client sites and so on was very helpful to us. The mechanics of working remotely were fine.
The issue was getting reliable audit evidence without being able to visit audited bodies. That is the bit I would have raised an eyebrow at in advance, but it did accelerate the digital exchange of information. We had to move faster than we were planning to on receiving date remotely, because we could not go and access that on site. That has been very helpful and is definitely one of the things we will be retaining.
We found that we could do audits remotely. We had always sent a team to stay away at the MOD in Bristol, for example, and they would be there for six weeks. We found that that was not actually necessary, so there is an efficiency gain. Going back now, we would go and visit for the opening to understand any changes, to meet the teams there and to assess for ourselves whether there are any new risks we should be worried about. But we do not need to be there for six weeks to finish the work, because we have the data exchange. We will go back and clear queries and so on. That is a more efficient use of our staff’s time, and it means a lower travel and subsistence budget for us, which is materially useful and more efficient.
Those are some of the areas we will hang on to. What we will not do is get used to never visiting our audited bodies. I think that would be very dangerous for an auditor. It is okay for one or two audit cycles, because you are using established understanding and relationships, but that deteriorates very quickly. A good auditor knows that they have to eyeball the people they are auditing in order to understand the body language, and they need to actually be around while the organisation is working. We cannot lose that entirely, so we are planning visits into the approach we are taking now. Kate, do you have any thoughts on this?
Kate Mathers: There are some very practical examples we have of things that we will take forward. I was talking to a team today who now have access to DWP systems remotely, which helps them enormously. The one thing we are also conscious of is that we are a training organisation. We have had two big intakes of new starters, who are learning to become auditors entirely remotely. That is a concern to us. We have a programme of re-engagement to make sure that our trainees are able to learn effectively with some face-to-face engagement with their teams and with clients.
Chair: We now have at least two—possibly three—votes in the House, so it is a neat moment to stop. We have a suite of further questions around this issue. I also wanted to bring Peter in to talk about remote auditing and the software you invested in some years ago, and we had one or two other questions. We are nearly done, but we are fortunate that we will be seeing you next week, so given the small number of questions—perhaps four—that we have left, I suggest that, rather than keeping you waiting for 30 to 45 minutes, we adjourn and continue next week for just a few minutes before we start our questioning on your annual report. Perhaps, CAG, you can cover for your two esteemed colleagues next week on these matters. I thank you all for coming.