International Development Committee

Oral evidence: The Independent Commission for Aid Impact’s Annual Report 2013-14, HC 523
Tuesday 8 July 2014

Ordered by the House of Commons to be published on 17 July 2014.

Watch the meeting: Tuesday 8 July 2014

Members present: Sir Malcolm Bruce (Chair); Fiona Bruce; Fabian Hamilton; Pauline Latham; Jeremy Lefroy; Sir Peter Luff

Questions 1-99

Witnesses: Graham Ward CBE, Chief Commissioner, Independent Commission for Aid Impact, Diana Good, Commissioner, ICAI, Mark Foster, Commissioner, ICAI, and Alexandra Cran-McGreehin, Head of Secretariat, ICAI gave evidence 

Q1   Chair: Good morning and welcome again.  We know who you are, but if you could introduce yourselves for the record, it gets it into the report.

Graham Ward: Good morning, Chairman.  I am Graham Ward, the Chief Commissioner of the Independent Commission for Aid Impact. 

Diana Good: I am Diana Good; I am one of the Commissioners.

Mark Foster: I am Mark Foster; I am one of the Commissioners.

Alexandra Cran-McGreehin: I am Alexandra Cran-McGreehin, the Head of the Secretariat. 

Q2   Chair: Thank you very much, and, as I say, welcome.  Sorry to keep you waiting a few minutes.  We know you have a division of labour, so all four of you do not have to answer all the questions—just as you think appropriate.  Obviously the relationship, both between ICAI and this Committee and ICAI and the Department, I guess has become more established in the last year.  You have had 12 reports, including the first red evaluation, so what do you take overall, both from the last year and perhaps the accumulation of what you might say about UK development that you have learned, or the particular comment you have that has come out strongly? 

Graham Ward: Well, thank you, Chairman.  What we have found compared with last year is that, in our work in those 12 reports, we have found more results at either end of the spectrum.  We have had two overall full green ratings, which, of course, means very strong overall performance; one overall full red rating, the TMSA, which indicates very poor overall performance.  Seven of our reports gave a broadly positive rating; the other five gave a broadly negative rating, indicating that there were issues that needed to be addressed. 

Perhaps I can look at trends in the context of the four different criteria that we use.  Objectives: we found examples of designs at the individual programme level that we believe to be innovative; they were well grounded in evidence as to what works, as the occasion required.  On the other hand there were examples where we were less convinced that DFID’s objective setting, in respect particularly of the more complex development goals, which required a portfolio of linked interventions, was well planned.

On delivery, we are concerned that DFID as an organisation as a whole is giving lower priority to the vital work of robust programme management and close oversight of delivery partners than it does to the more cerebral processes, such as programme design.  We were pleased to see improving programme management as a departmental priority. 

On impact, we had eight green or green/amber scores out of 12, which we think is very creditable.  We think that provides a measure of assurance to British taxpayers that UK aid is making a real and a positive difference to intended beneficiaries.  Nevertheless, impact could be higher if shortcomings in programme objectives, delivery and design were addressed.

On the learning side, and the scores on learning were the least positive overall, we did become concerned over the course of our reviews that the experience that DFID gains from the programmes that it carries out is not being captured in a systematic way.  It is not being shared across the country programmes, and therefore we commissioned a special report on learning in order to explore the roots of the problem in more detail, and to understand how DFID learns and how that learning process might be improved.

Q3   Chair: We got that from the report.  There are two things I would ask there: how close were the amber/reds to being red?  They were quite critical; what stopped you making them red?  Given that you did these two thematic reviews on learning, and also on the private sector, was that prompted by a concern that on both of those issues you decided to do a thematic report, because you did not think from what you had seen that DFID was as strong as it should be? 

Graham Ward: On your first question, in terms of how close were the amber/reds to red, I do not believe that in any of those cases we went through agonies in terms of which of them it should be.  On the learning issue, I will ask Diana to expand on that in a moment, because she led on that report.  The reason that we looked at private sector was because it is a big new initiative; it had been given a lot of weight by DFID and the Secretary of State in speeches.  We thought, “This is a major area, and it would be good to look at that right at the very beginning.”  I will ask Diana to talk about learning in a little bit more detail.  

Diana Good: Red is clearly a very serious rating indeed; we would reserve that for programmes and work that we look at that we feel really falls very short indeed.  In the case of the TradeMark Sothern Africa, TMSA, report—which, of course, you have also looked at very closely—there were some really very serious deficiencies to the point where we felt that this was a programme that did need to be looked at extremely carefully.  Indeed, we entirely support the Secretary of State’s decision to cease funding and do a very full review. 

In the case of the amber/reds, as Graham said, it was not a finely based judgment as to whether these should actually be red.  Again, we regarded these as being areas that did have areas that needed serious attention, but not to the extent of the kind of thing that we saw in terms of governance failures in TMSA. 

On learning, we felt that there the biggest issue was the level of inconsistency, and the failure to have coherent, systematic processes and incentives in place to ensure that DFID staff across the board, as an organisation, learn as well as they should.  There it was a story of differences: some very poor learning results, some of which were red, by way of example Nepal, and others which were very good.  The issue there was: why is there that level of inconsistency?  Why are there not sufficient processes in place to ensure that the organisation as a whole learns as well as some individuals do in different parts of the world?

Q4   Chair: That was the point I think you made: that the individuals could be great at learning, but the institutions do not always—

Diana Good: Exactly, and so it is very dependent on how good the head of country office is, in terms of encouraging learning within a given country or programme. 

Q5   Chair: Two final questions: one is, on the back of that, how much do you feel you have added value for the taxpayer?  You say with your memorandum of understanding that you do not do policy—you say that is for us or the Department—but is that not a slightly arbitrary division?  Inevitably, when you are looking at the operation of a programme, you have to know what the policy is behind it to determine whether the programme is actually delivering, so should you be quite so coy?  

Graham Ward: Let us take the second question first; that we should not look at policy was one of the givens when we were established in the first place, and it was even explained to us at one point, by somebody in DFID who was setting us up, that it would be something of an arrogance for us to talk about policy, because that was the role of the Committee and it was not appropriate.  What we hope we have been able to do is to provide you with strongly evidence‑based reports that enable you to form views on policy that hopefully are better informed than the views you were able to form before we were put into place. 

We do believe that we have produced value for money for taxpayers, and improved the value for money and the effectiveness of DFID through our work.  We think that the implementation of recommendations that we have made has really enabled more beneficiaries to get more benefit out of our aid work in UK than was the case before.  We have been able to give clear advice that has resulted in better value for money being delivered.  The TMSA report is perhaps the major example during the course of the last year, where some £70 million of British taxpayers’ money that was languishing in a bank account in Southern Africa, producing no good for anybody, as a result of the report that we issued was able to be released and is now able to provide some benefit to the poor in the world.  We think that is something that is very worthwhile. 

Mark Foster: The follow‑up process this year was a more detailed process, looking back to the work we had done the year before and understanding the extent to which DFID had followed up on our recommendations and were changing.  We were generally encouraged by the fact that in many of the cases there was direct evidence of the fact that they had taken on board our recommendations and were changing things substantively.  We obviously had some areas we would have liked to have seen some more done on, but the overall sense we had was that they are taking our role seriously and are responding to our recommendations, and that has an overall effect of improving the system.

Q6   Fiona Bruce: As part of your follow‑up of the Year 2 reports, you assessed DFID’s relationship with the multilaterals.  What did you find?

Graham Ward: In the work that we looked at in respect of the multilaterals, we have continued to find that the way in which DFID monitors the relationships and takes part in relationships is not as active as we would like to see.  They tend to think of multilaterals more as development partners than as people who are implementing UK development projects, even though, in many cases, in effect, the multilateral has been hired as a delivery agent.  We think there is tighter management that can be brought in, as far as that is concerned. 

Also we have found that the MAR, which assessed the relative merits of different multilaterals, was based far more on policy assessments and on central issues, as far as the organisation of the multilateral was concerned, than it was on the ability of the multilateral to actually deliver on the ground in individual countries.  Therefore, we have recommended strongly that much more effort should go into assessing the delivery capability on the ground in a country before going nap on a particular multilateral.  Diana has some points to add on this as well.  

Diana Good: To take, by way of example, UNICEF, we have done a follow‑up on our report on UNICEF and also a follow‑up on our report on education in Nigeria, where UNICEF is one of the delivery partners.  Both of those follow‑up exercises have been very interesting.  In terms of the central relationship with UNICEF and the concern that we had expressed about oversight, significant efforts have been made by DFID to improve that oversight and relationship, and they have put new mechanisms in place: a portfolio delivery review system and a framework arrangement.  DFID feels that is enabling them to tighten up their supervision of what UNICEF is doing.  That is at a central level.

Our concern as to the ability of the multilaterals to deliver consistently and well at country level remains, and our follow‑up review of education in Nigeria absolutely illustrates that point.  There were two very differently performing programmes, one run by Cambridge Education, ESSPIN, and one run by UNICEF, GEP.  There was a very stark contrast in terms of performance, and we were very concerned about the serious deficiencies in terms of the inadequate processes and systems in place applied by DFID to UNICEF. 

Now three-quarters of DFID’s funding in-country goes to UNICEF, so there is a very real issue in terms of whether a now improving oversight at central level is actually feeding through to the culture in terms of oversight in-country.  For there to be two programmes in Nigeria, both of them trying to achieve the same things in terms of education, one performing so much strikingly better than the UNICEF one, indicates that there is still an issue.  We recommended some very serious changes to the UNICEF programme and to the overall programming, and it is with considerable regret that we observe that it took nearly a year for DFID to decide to put the UNICEF part of the programming on to a programme improvement plan. 

It took them a year from the publication of our report to take that action.  Now that PIP was due to report at the end of June, so at the end of last month; we, yesterday, in our meeting with management asked what was happening, and they said they did not know whether the report had yet come through and it would take them some months to absorb it.  We feel very strongly, in view of the report we did on education in East Africa, where we felt that enrolment was far too much the goal and there was not nearly enough emphasis on learning, our earlier follow‑up on education in East Africa indicated that there had been a complete new take on the attitude to learning in East Africa.  They have put together a good position paper, and their focus on learning matches very much the ICAI framework.  In Nigeria we still see this contrast, so there is a question as to whether, in terms of culture and oversight in-country, a multilateral like UNICEF is becoming almost too big to fail in DFID’s eyes.

Q7   Fiona Bruce: That is very interesting.  Are you going to follow up on this quite closely?  In what way, and will it be with other multilaterals as well?

Diana Good: Mark is going to be lead commissioner on the report we are doing on multilaterals.  I want to say, in terms of follow‑up, we are intending that in this next year we will do even more intensive follow‑up than we have done up until now.  It seems to us it is really fundamental.  In the Nigeria report we identified that UNICEF’s performance would have, by itself, merited a red/amber, and then it has taken over a year for any action to be taken.  We think this is a programme where there is a very serious question as to whether it should not be terminated.  It demonstrates that unless we do very full follow‑up, the recommendations we are making may not be making the difference to the poorest people in the world.  In the context of Nigeria, and education and girls, it does seem to be peculiarly acute.

Graham Ward: May I ask Mark to explain what we are going to be doing specifically on multilaterals in the current year?

Mark Foster: Yes.  One of the major thematic reports this year will be on the role of multilaterals in DFID’s portfolio; it is taking place at an interesting time in terms of how they are looking at the use of multilaterals, both in terms of their engagement with the centre but also their role, as we have heard, in-country, to try to understand the extent to which their engagement, particularly with the core funding processes as much as with the bilateral processes, really are joined up.  That is probably going to be the key area we are going to look at, because, to some extent, we have seen a fair bit of interaction now at country level in bilateral programming with these various agencies, and we understand that dynamic quite well. 

We are now trying to understand more the relationship of governance through the core programmes, and connecting the dots between those two pieces, and understanding a bit more about the role that the UN multilateral agencies in general play in the portfolio compared with NGOs or contractors as the delivery vehicles.  That is particularly in the context of fragile and conflict states, where often the multilaterals are seen as a default solution, and that can lead to some relatively poor performance sometimes.

Q8   Chair: We had evidence previously saying that the BAR was very good, the MAR was very good, but there was no trade‑off between the two to determine whether bilateral or multilateral is better way of delivering a particular programme.  Presumably your report is going to assist in that?

Mark Foster: Exactly, and frankly we are getting some very interesting evidence from the report I am currently doing on scale‑up of fragile states’ funding as well, because it is often that we are seeing very live examples of the way that multilateral agencies play a role in those dynamics.  I think we are going to have a very clear amount of things to say about that.

Q9   Jeremy Lefroy: I would just declare an interest as Chairman of the Parliamentary Network on the World Bank and International Monetary Fund.  As you are probably aware, the World Bank is going through a major reorganisation at the moment, concentrating more on themes rather than on involvement in specific countries or its country programmes.  I can well understand the reasons for that, but have you come across any influence that DFID would have had in that, or has that just happened without really any input from DFID or, indeed, any other contributors to the World Bank, given that the World Bank is one of the major recipients of DFID funding?

Graham Ward: We have not specifically looked at that decision‑making process as you set out there, Mr Lefroy.  We have seen the relevant department of DFID do well in terms of improving the performance of the Asian Development Bank, and there they did exercise a lot of influence, in a good way, in terms of improving decision‑making processes, improving monitoring, and giving better guidance to their offices around the world in terms of dealing with the Asian Development Bank.  I have got no personal knowledge of how they have interacted on this World Bank issue; it is not something we have studied specifically.

Mark Foster: We have not looked this particularly; I think it would be one of the things we want to look at in our multilateral review—that kind of influencing process.  What is important is to see what the balance is between central influencing over things such as the results frameworks for these agencies, which I think is good, and influence over reform processes, and what they are trying to do to make them operate more jointly on the ground in-country, as a coherent body of the international community trying to make a difference on the ground.  That, to me, is another area where the co-ordination of activity and, frankly, division of labour and clarity of roles among the players needs to be increasingly clear, because I think that host governments are confused by some of that.  It is important that there is an overall sense of what the architecture is.   

Graham Ward: The multilateral study is at a very early stage, so we will be able to make sure our work can cover the point that you have raised. 

Jeremy Lefroy: Could I just urge that that happens?  It is very important.  It has been very controversial—as I say, there are probably some very good reasons for it—but there was a visceral attack on it in the leader of the Financial Times recently, so it is very important that it is looked at, whatever conclusion you come to.  Thank you, Chair. 

Q10   Pauline Latham: In your Year 2 reports, you said DFID should strengthen learning from contractor‑delivered programmes and from NGOs.  Has that improved and should there be more involvement of DFID advisers in-country in managing contractors? 

Graham Ward: Mark led on our contractors report, so I will ask Mark to start on that one, please.

Mark Foster: I think that one of the positive reactions to our contractor report was some further reinforcement in DFID of what I call their key supplier network with regard to the players in this contactor space.  They are bringing them together more frequently, and there is therefore evidence of some, if you like, high-level knowledge sharing starting to occur there, and also I think a more structured programme around how they are thinking about them from a procurement process.  I still think there is a lot of work to do around the overall programme management structures that DFID applies to its use of contractors in-country. 

I was encouraged to see the focus on programme management in the recently published Smart Rules.  The question mark will be now around how they end up being truly implemented and whether there can be sufficient capacity of both context and content-aware programme managers who can be in the field close enough to the action to really provide appropriate scrutiny, or that there can be really clear line of channels of command and operating models between DFID and the contractors on particular pieces of work.  That is going to be one of the critical learnings over this next couple of years as those new processes are rolled out. 

I am pleased to see that there is now as much focus hopefully on not just programme set‑up but programme delivery, and that will be the critical thing: can the new structures allow DFID to keep its eyes on things during delivery?

Diana Good: In our report on learning, we emphasise this point, but there needs to be far better and more consistent learning by DFID from its partners, from its contractors, from its implementing agencies and from the beneficiaries themselves.  The critical thing is a question of ensuring that they are learning in real time during implementation.  There is a lot of emphasis on the upfront work in terms of producing a business case, and certainly there are many evaluations carried out after the event, but it is this issue of learning from the people who are implementing and the people who are affected—the beneficiaries—and their representatives in terms of CSOs.  We have seen pockets of excellence; for instance, in the PPA report we were most impressed by the learning partnership that had been created.  What we want to see is this being identified as a pocket of excellence, but then replicated across the board.

Q11   Chair: Does that require some organisational change—

Diana Good: Yes it does.

Q12   Chair: —a unit of some kind that draws all these threads together?  When we did our report in Brazil, where DFID does not have a bilateral programme, part of our concern was whether innovative things happening in Brazil for urban poverty reduction could inform DFID in Africa, for example; we were not clear what the mechanism was for picking that up. 

Diana Good: No, they are not clear about it.  I would just emphasise that the default position should not be, in our view, to create another massive impenetrable database.  Data can be pretty impenetrable in its own right; it needs synthesis; it needs interpretation; it needs communication by the people who know what they have seen is working well or not working well across the board.  It is not to say that there is no communication; they have a cadre of PSDs; they are meant to spend 10% of their time, and they obviously value what learning and sharing of learning comes out of that.  However, relative to both the task of communication and the importance of it, it is not nearly enough. 

Q13   Jeremy Lefroy: I think we have mentioned this to DFID on at least two occasions, if not more: the importance of learning and collecting that learning, both for use within DFID itself and potentially for use within the Government more widely.  There are things that DFID is doing in many countries that civil servants and others in this country have no exposure to and that I believe they can learn from.  As far as I am aware, there still does not seem to be any mechanism or any ability to get that across into other parts of Government.  I may be completely wrong on that, and would welcome you contradicting me. 

Diana Good: Clearly, there are areas where they are working together—the conflict pool is a case in point—but there are challenges, so I would agree with you entirely.  To extend the point, which I think also is a very critical point, it is sharing that learning with national governments and, indeed, with other agencies.  This is a global effort, and DFID is a very influential and important player, but it is only one.  Knowledge is fundamental here, because if there are parts of Whitehall that are not as familiar with what DFID is doing that is working well and not well, what are the challenges for governments in particular in conflict affected and fragile states, and even middle-income countries?

Q14   Jeremy Lefroy: It is one further argument for our commitment to ODA that that learning can help in this country as well as elsewhere.

Diana Good: Absolutely.

Graham Ward: Yes, indeed.  It works two ways; Mark in particular will have seen learning that is gained by contractors, for example, during the course of doing work for DFID is not as sufficiently systematically gathered and shared with DFID as a whole.  There are pockets of really useful stuff that are off in a siding somewhere, rather than being part of the core of the thinking and knowledge of the Department as a whole. 

Q15   Sir Peter Luff: This is a very straight question, I promise you, with no hidden agenda.  How confident are you that your work programme for this year reflects your strategic objectives, is coherent with the rest of the workings of Parliament, and meets your mandated responsibilities?

Graham Ward: Well, we put a lot of work into putting together the work programme.  We have had discussions with the Committee.  We have had discussions with the Secretary of State and with DFID officials; we have had a public consultation in respect of the programme that we are putting together. 

Q16   Sir Peter Luff: On that, how many responses did you get to the consultation?  How well received was that?

Graham Ward: I cannot remember off the top of my head.  Can you remember? 

Alexandra Cran-McGreehin: I do not know in detail. 

Graham Ward: It was several dozen I think.

Q17   Sir Peter Luff: A good number.

Graham Ward: Yes.

Q18   Sir Peter Luff: From aid agencies, typically?  NGOs?

Alexandra Cran-McGreehin: A mixture; yes, we had NGOs; we had think tanks; we had some individual members of the public as well, so there was a bit of a mix. 

Graham Ward: What we then did was try to balance all of those things up together with what we have looked at in the past to try to get a proper measure of coverage, so we have got a proper attention to materiality.  Then, of course, Chairman, we came back to the Committee and said, “These are our proposals.”  We had some further interaction between us before the Committee approved the programme that we put out for the year ahead.   

Q19   Sir Peter Luff: You are happy with the programme.

Graham Ward: We are happy with the programme, yes. 

Q20   Sir Peter Luff: What is the most important thing you are going to do this year?  What is your flagship?

Graham Ward: Up the top of what we are going to do this year are the multilateral review, which Mark has addressed already, and the review on impact and DFID’s approach to impact, because, at the end of the day, this is the thing that is bringing benefit to the poor in different parts of the world.  Diana is going to lead that, so I will ask Diana in a moment to give a bit more of a flavour for what it is we are going to do in the impact study.  Indeed, we have got a meeting with stakeholders, which is taking place this coming Thursday, and, as part of that, we are going to be taking their views on the whole impact question, and we will incorporate those into the work programme that we put into that study.  Diana?

Diana Good: I hope that one of the things that will be striking about our work programme this coming year is that there will be cross‑cutting themes that emerge across the body of them.  I think they are going to build and complement on one another, whereas in previous years we have pointed the lens in a series of directions.  By the time we are here next year, we will be looking at a combination of the security and justice programme that I am going to be leading on, the anti‑corruption one that is just being completed, and the scale-up in fragile states. 

We will combine that together with the impact report—we have deliberately delayed the impact report so that we have the benefit of these other studies and reviews—so that the impact report can be, to some extent, a reflective piece on what will be the body of 40 reports that we will have done by then.  That will allow us to really scrutinise and identify what we feel from what we have seen is working best and why, and if that is replicable across sectors and across countries, whatever the context, and to look very hard at what DFID thinks impact is, how it sets about achieving it, and what processes and tools it has in place to seek to maximise real impact, making a serious beneficial difference to the poorest people. 

The aim is that the work programme will hang together, and all of these pieces, all of them very significant pieces of work, will feed and inform one another.  We are ensuring that there is a very close liaison between the teams, and obviously there always is between the Commissioners, in terms of ensuring that this body of work produces a coherent body of thematic issues.

Q21   Sir Peter Luff: In that search for coherence, how many of those reports have you actually begun work on? 

Graham Ward: We have started work on all but a couple, and even in relation to the couple we have done some very preliminary scoping work, so there is nothing in there that is hanging in the air.

Q22   Sir Peter Luff: Is it going well?

Graham Ward: I think it is going well, yes.  I think it is going well.  I believe we will be able to publish some stuff that will be of real value.

Mark Foster: The point that Diana has made about the coherence of the piece is a really important one in terms of quite a difference in terms of how the pieces fit together this time around.  I think that, given the fact that 30% of the funding is planned to go to fragile and conflict stages, the scale-up report has an opportunity to have some very important thematic advice around a very big part of where a lot of the funding focus and, indeed, the energies are going at the moment.  That one is also drawing a number of threads together as well, so hopefully that will also be a particular piece in the story this year. 

Q23   Chair: Will that help or dramatically alter the way DFID measures its own impact?  Obviously they have been criticised for being too results focussed, the argument being that impact cannot just be measured in results and some things take longer than others.  Are you going to be looking at that multidimensional approach and what impact is? 

Mark Foster: Absolutely.  That is at the heart of the report that Diana is driving, but it is drawing on a number of themes we are seeing.  When you put something, as I have been recently, both in Somalia and DRC, and try to put the results framework as being the objective for those two things, you see question marks about fitness for purpose for that to be really the driver of what you are trying to do.  That is where I think we need to be more thoughtful about the really appropriate goals to have in those kinds of environments, with the right level of ambition and realism, and then drive towards them.

Q24   Sir Peter Luff: What difference do you feel you have made?  How much more confidence is there in DFID’s work as a result of what you have done?  How much more politically robust is the commitment to international development?  How do you measure or assess that?

Graham Ward: The measure of it has varied from report to report, so in terms of making a difference, if you look at one end at TMSA, there was clearly something that really was not working at all well, the programme was closed down, and funds were released for better use elsewhere.  The Secretary of State initiated a programme to make sure that internal programme evaluation by DFID would happen more on time and more frequently to increase the effectiveness of the internal audit department, so there was a range of results that came out of that.  If you go back to one of our very first reports, which was looking at anti‑corruption, we have seen DFID introduce new processes and approaches in terms of the proper use of its own money, as well as encouraging other governments and helping them to fight corruption within their own territories.  We have seen some real advantages there. 

If you look at some of the stuff that we have done in relation to health, we were able to advise on efficiencies and on better negotiation of contracts with implementing parties.

Q25   Sir Peter Luff: Do you feel this cuts through beyond being a specialist model to more general public awareness of the improved scrutiny of DFID’s work?

Graham Ward: One of the things that struck me right at the very beginning, when I first came along and met the Committee for my pre‑appointment hearing, and it has stayed with me as a theme all the way through, Sir Peter, has been that DFID does not make enough of the successes that it has.  There are a large number of really very useful and potentially very bright lights that are hidden under a very large number of bushels.  I really think that they could do far more.

Q26   Sir Peter Luff: You should give them the confidence to be proud of what they are doing.

Graham Ward: Yes, I think that is exactly right.

Mark Foster: I would add one more comment in terms of the impact that you see, which is actually on the people in the field, because it is now very commonplace that people you bump into reference our reports directly in their conversations.  People inside DFID are using our reports actively to change the way they programme and the way they think.  It is almost that you cannot really sit in a meeting now anywhere across the DFID countries I have visited, without them saying, “We used the Horn of Africa for that.  We looked at that one there for child mortality.  We liked the thing you were doing around procurement for contractors.”  The awareness of what we are doing is really quite interesting as far as how they are using it as an internal tool.  I think that they are on their toes more as a result of this.

Q27   Chair: It is a refreshing comment on DFID, both for you and for us as a Committee, that they are very well aware of how much potential criticism is out there, they are very committed to what they do, and they do actually listen to the evidence that is presented to them.  They have their own ideas, so there is obviously a creative tension, but it is a fair comment.

Diana Good: I would say I think the points that we have been making are absolutely underlined in the learning report, which is that if DFID does not have the systems and culture in place to ensure that its best practice is well known through its own organisation and, indeed, across Whitehall—and, indeed, with national governments and so on—then we are able to shine light on that, because are looking across the board.  As Mark has just described, it happens every time we are in-country; it is perfectly clear.  Of course people in country offices are nervous about the scrutiny, but the feedback we consistently get is that they find the work we are doing really very helpful. 

I just want to add one last comment, which is that when the Permanent Secretary was asked this question a year ago, he said that he thought we had made a significant difference in terms of DFID’s focus on the beneficiary.  If we have made that difference, and that is at the forefront of everything they do, that is something we can be very proud of. 

Q28   Fabian Hamilton: You can imagine that we are very keen to ensure that there is the smoothest possible transition to the next phase of ICAI, for Commissioners and staff, next June, 2015.  I wondered whether you could update us on the next phase following the current mandate’s expiry in May of next year.  How do you think the organisational structure might change, and what planning is being done to ensure that smooth transition?

Graham Ward: Alexandra is the continuity across that divide, so perhaps I could ask Alexandra to lead into this. 

Alexandra Cran-McGreehin: Yes, thank you very much.  As you know, because of the way ICAI was set up, we were set up for this four-year period, so everything is set up just to expire at the one time, which is not ideal obviously.  There is a significant amount of change and large turnover of personnel that we are going to have to deal with; on the commissioner side, all the commissioners will change during the first half of next year.

Q29   Fabian Hamilton: There will be an overlap period, I presume.

Alexandra Cran-McGreehin: The new chief commissioner, as you know, is currently being recruited, and should be in place by the end of the year, so there will be some overlap with the current commissioners there, and one other commissioner is being recruited on a similar timescale.  The other two new commissioners are planned to be quite a bit later into next year, so there is not so much overlap there.  Even on the secretariat side, all the current staff bar our executive assistant are all due to leave in the coming year; there may be ways that could be mitigated, but as it stands, because they are not permanent posts, everyone is moving back to their Departments. 

Obviously our contract with our current contractor consortium ends next May as well; the procurement process for that is under way in its early stages, but you do have everything changing at once, so there is a really high risk, for us, that our ability to carry out our mandate is pretty disrupted.  There are two particular issues; one of them is our ability to continue to deliver reports in a smooth process.  I think we are realistic that we will not be able to deliver the same volume that we were delivering in the last couple of years, just because of all that change going on, but we feel we have a public and moral duty to keep producing reports at a decent rate, so we want to explore with DFID a bit more ways we can continue to do that next year post‑election. 

Q30   Fabian Hamilton: Sorry to interrupt you, but it sounds, Alexandra, as if there may be some disruption and it will not be that smooth.

Alexandra Cran-McGreehin: I think that is fair.  We flagged in our annual report that this was a high risk and that even with mitigating actions it remains a high risk for us.  There are things that we can do, and we do want to continue to talk to DFID, particularly around this handover point; we need to make sure that there is sufficient handover at all levels to make sure that all the learning from the last four years is not lost.  ICAI was set up as a new body and we have learned a huge amount about how to do what it is we have been asked to do, and we want to make sure that that is not lost and that the work continues to expand and develop. 

You also asked about structure; we have also been reflecting on how well ICAI worked in the first four years, and we are in the final stages of discussing with DFID just a bit of a rebalancing.  At the minute, we have a contractor management function, which is designed to programme manage all the different review teams we have.  Quite a lot of that also happens in the secretariat as well, so you can imagine that, even if you do that in the most efficient way possible, there is a bit of duplication going on there, and so we believe that it would be better to have a larger secretariat and remove that contractor management function, so that you have everything happening in one place, and that would create some efficiencies.  It would also improve the resilience of the team, because what we have found in practice is that very small things can really disrupt the work of a team of five, because you do not have enough people to manage some difficult circumstances. 

That is our plan; we discussed that with contractors at an early market engagement event.  It was very good for us that there is a lot of interest out there—we had a high attendance at that event—and we discussed these ideas informally, and also more output-based contracting ideas, so trying to figure out how we can get the best value out of the next contract, so that we can deliver even better value for the taxpayer with our reports. 

Graham Ward: Sorry, the resilience point is hugely important.  Just to give you an example: of the five staff, one has been called to go on jury service in a few weeks’ time, and another one has been recruited by DFID to go overseas and carry out a role for them.  All of a sudden 40% of our resource has disappeared.  We have all the management challenges of an SME here, which is very disruptive.

Q31   Fabian Hamilton: I gather you are going to be asked to vacate your current premises in Whitehall.

Graham Ward: That is a real risk; we have not been served noticed at the moment, but there is a real risk, because the Cabinet Office has decided that they might wish to use the space for something else.

Q32   Fabian Hamilton: What plans have you got for relocation, because that is going to be pretty serious too, is it not?

Alexandra Cran-McGreehin: We explored a number of options; we looked across the Government estate, but there has been a lot of rationalisation in the last few years, so there is nothing available, even for a small team like us.  It boils down to private accommodation.  We also explored this with DFID, and as of yesterday they have committed to us that we can work from 22 Whitehall in future.  There are obvious questions of perceptions of independence that we are very aware of with that.  I think that it would not have been a good idea at the beginning clearly, but we feel that we have demonstrated very strongly our independence for the last two years, and that independence is far more about how you act then it is about where you are based, but we just need to be very alive to that.  We have also been discussing with DFID that it is very important how those arrangements would work in practice; we would need to have private space, so that we can have free and frank discussions, and all of those kinds of things will need to be sorted out appropriately to make that work.

Q33   Fabian Hamilton: You are confident that will not compromise your independence.

Alexandra Cran-McGreehin: I do not believe it should, no. 

Graham Ward: Independence is a question of personal moral courage as much as anything.  I certainly believe that this team here has got that in great strength, and that is what really counts. 

Q34   Pauline Latham: DFID spends huge amount of money on self-evaluation, with £200 million committed at the moment.  When you are coming up with your reviews, do you draw on their work, and can you give us some specific examples if you do?

Graham Ward: Yes, we do draw on the work.  At the earliest stage of carrying out one of our reviews, we look at other work that has been carried out in the relevant area; we look at work that has been carried out by this Committee, by the NAO, by DFID itself, by anybody else that might have had a look at what is going on, and we really make sure that we get that all built in, so that we can take account of what is there.  I know Mark has been studying this in a little bit more detail. 

Mark Foster: First of all, whatever evaluations are out there are required reading, I guess, before we start anything we are doing.  Also on the ground we make a point of meeting any M&E-related staff that are involved with the programmes we are looking at.  For example, I spent a long time discussing the M&E around the anti‑corruption programming in Nigeria just a couple of weeks ago with the lead evaluator there, so that we can understand what methodology they are using, what approaches they are using, and what they found so far, so we engage with it.  I think it is fair to say that the scale-up of embedded M&E within DFID is still going on; the rollout of what work is starting to come out in terms of outputs that we can use is also at a relatively early stage, so I expect this to be a much bigger part of the next phase, where there will hopefully be a richer body of that work to look at.  We certainly engage with whatever there is now as part of our process.

Diana Good: If I can add to that, the best example of where we have used an existing DFID evaluation and looked back at it subsequently, in terms of whether it has had a lasting effect, was our report on livelihoods in Odisha.  For the purposes of the impact review, we very much had in mind that we would do an Odisha on other similar reports, but centrally DFID has not been able to point us in the direction of any equivalents of the Odisha evaluation; they just have not done them yet.  They may have been done in-country, so we are now working with DFID to identify whether there are country evaluations that have been done that the centre was not aware of, but the likelihood is that that embedded evaluation has not had long enough in action to generate that kind of work yet. 

DFID’s stated intention is that 28% of future work will be impact evaluations, but if those have not yet been done it is not something we can take into account.  I would emphasise what Mark has just said: in the future that will be a rich source of information for us to review, but if it does not yet exist we obviously cannot take it into account. 

Q35   Pauline Latham: Is £200 million value for money?

Graham Ward: I think the simple answer is that it is far too early for us to be able to form a view on that. 

Diana Good: What we said in the learning report was that DFID does not have in place a process for scrutinising, reviewing and learning from whether its evaluations are making a difference in terms of impact, so it is a question mark.

Q36   Fabian Hamilton: An evaluation expert commented that “ICAI needs to do more to recognise DFID’s aid interventions usually depend critically on actions by others, recipient countries and other donors”.  I wonder if you would agree, and whether your recommendations need to be relevant to other stakeholders, as well as DFID, and, if so, how do you plan to do this?  

Graham Ward: Certainly we would agree with the point that successful implementation of an intervention depends not only on what DFID does but also critically on what is done by the host government, and also critically on the performance of those who are appointed in order to implement it.  I do not accept that we ignore that in the course of our reviews; I think we have paid a great deal of attention to all of those factors.  If you go back to look at, say, the work that we did on Odisha, we commented on and studied in some detail the interaction between DFID and the implementers on the ground, the state government of Odisha, and the way in which that facilitated and, indeed, helped to design the programme there, and then the involvement of the Government of India as a whole in that. 

We always look at all of the players there and how they all can operate together.  When it comes to recommendations, then clearly our remit is to make recommendations to DFID, and so the style of a recommendation that we make will tend to be that DFID should interact with, shall we say, a host government in order to improve some particular facet of an intervention or a programme.  I do not think we have the locus to give direct advice to sovereign governments overseas in terms of how they should go about their business.

Q37   Fabian Hamilton: Some of the witnesses in both last year’s and this year’s Inquiry suggested that there should be joint evaluations with other donors.  Was that something you would consider doing?

Graham Ward: It is actually, yes.  We have considered it pretty well from the very beginning.  One of the first meetings I had after being appointed was with the Chair of UNDP, and we went quite a long way down the road in respect of two or three possible joint evaluations.  Where we have come up against barriers to doing it is that we generally work much more quickly than other evaluators, so that would mean that, if we were to work according to their timetable, our ability to report promptly to you, and to give advice promptly for DFID, would be impaired. 

Also, most of the others negotiate their conclusions and recommendations with the people that they are reviewing.  As you know, we do not do that; we tell you the straight truth about what we believe our conclusions should be and what our recommendations are.  That qualitative philosophical difference of approach has caused an issue as well.  What we do do is meet with these other people, so during the course of the last few months Alexandra has been to an OECD DAC meeting and has met a large number of evaluators there. 

I have had personal discussions with the American agency that evaluates the work of USAID in Afghanistan.  I have had personal discussions with government officials and representatives from Australia about what they do in terms of their evaluation.  We are in touch, and we are aware that we have got potentially things to learn here—there are benefits from us sharing our approach and lessons with these people—but regarding the actual joint evaluation, I am afraid there are these very important barriers that have come up.

Q38   Fabian Hamilton: Does that procedural and philosophical difference happen with every other evaluator and every other donor organisation, or is it just the majority?  Are there any that work like you?

Graham Ward: We have not yet identified any others that work in the way that we work.  We have identified some people in overseas countries who have said, “We only wish we could have our people work like you work,” but that is getting into the processes of overseas sovereign governments again, so they have undertaken to go and have their own discussions in their own countries. 

Q39   Chair: Thank you very much.  Before we finish, is there anything you want to tell us?  Do you get all the co-operation you need from DFID?  You have said you do not think they always follow it up as well as you would like, but do you get the co-operation you need?  Have you come across any barriers or any negatives in your operation that get in the way?

Graham Ward: The co-operation that we get from DFID officials, both centrally and in-country, has been good, Chairman, so that continues to be the case.  Indeed, that has been true where we have looked at other Government Departments; we have had a good level of co-operation there, so that has been positive.  We have discussed in earlier meetings with your Committee the question of the responses and the inconsistencies in management responses, particularly where they are already partially doing, or have started after our field work, things that are the subject of our recommendations.  We are having more of a discussion with DFID about that.

Q40   Chair: We have had some issues; are they volunteering all the relevant information that you need when you are in an inquiry, or do you find, as we have occasionally found, things that were not volunteered that you found out later?

Graham Ward: It is fair to say that some volunteer more and more promptly than others.

Mark Foster: I think it is fair to say that the quality of briefing materials that we are now provided in terms of the, if you like, starter pack we receive has got stronger, and we are seeing more of that without having to dig it out.  The starter pack is strong; still obviously you find things out as you go around, which is as much about the learning process and what people know exists as about motivation for sharing. 

Q41   Jeremy Lefroy: Obviously we have already discussed TradeMark Southern Africa and the £70 million that was sitting there.  I think ASI in their evidence questioned whether there were other such instances of money simply lying there.  I recall a couple of years ago some concerns about the Asian Development Bank: a transfer, if I recall rightly, of quite a substantial sum just before the financial year end, and the money sitting around for quite some time.  I wondered if you had made enquiries as to whether there were other substantial amounts of money sitting in accounts that had not been used.

Graham Ward: We have raised this issue with the Executive Management Committee in DFID.  They have assured us that they have done a search for such things and not come up with any, but I suppose, if it does not sound a bit cheeky, there will perhaps be an opportunity for you to ask them in a moment as well. 

Q42   Chair: Thank you very much again for that, both for your report and for giving us the evidence today.  I think we can honestly say that three years down the track now the workings between us and you seem to be very complementary, and you obviously are able to do things that we cannot do, but then perhaps the reverse is also the case.  We have also fed into each other fairly constructively; I hope you think that is also the case.  We have not had any serious hiccups.  The odd issue that has come up we have managed to resolve, so thank you for that.

Graham Ward: Many thanks, Chairman.  Yes, I think we agree.

Q43   Chair: Can I just suggest if you do move into offices in 22 Whitehall, can you put the “independent” on your door?

Graham Ward: Absolutely. 

Q44   Chair: It is probably just as well you are moving out of the Scotland office; they might get confused.

Graham Ward:  Perhaps we should change our house colour to blue, Chairman.  

Examination of Witnesses

Witnesses: Richard Calvert, DG Finance and Corporate Performance, DFID, Nick Dyer, DG Policy and Global Programmes, DFID, and Pauline Hayes, Acting Director, Western Asia Division, DFID, gave evidence.

Q45   Chair: Thank you very much for coming in.  I know you only came in for the tail end of the last evidence session, but you obviously know what is in the ICAI report.  Again, we know who you are, but if you could just introduce yourselves for the record.

Richard Calvert: Of course, and apologies on behalf of Mark Lowcock, who would normally have been here today but I think you know is not available.  I am Richard Calvert, the DG for Finance and Corporate Performance.  Nick Dyer is DG for Policy and Global Programmes, and Pauline Hayes is the Acting Director for Western Asia.

Q46   Chair: Thank you very much.  Perhaps starting at the absolute nitty-gritty, you got two green ratings from ICAI during the course of the year, but you also got your first-ever red, which was obviously the TradeMark Southern Africa issue.  You also got four amber/reds, which had some pretty substantial criticisms of your work.  Do you think ICAI is getting more critical or more confident, and how does it affect your relationships?  It is independent, so it will say things you do not like.  Does it have an effect on the relationship?

Richard Calvert: I do not think so.  Obviously, we have welcomed the scrutiny that ICAI gives us.  It was an important part of our process of driving up value for money and challenge and scrutiny over the last three or four years.  Over that period, ICAI has learnt much more about the Department.  It has continued to develop its methodology; it is looking at more areas. 

Although it is true that this year we have had, perhaps, a wider spread of traffic light scores, if you unpick that, then they fall into some slightly different categories.  Clearly TMSA was a specific project where things had gone wrong.  You, as a Committee, have had a detailed hearing on that.  We have tried to use that in a very positive way to learn lessons for the organisation as a whole.  We have done a lot of work to strengthen our programme management capability and processes in the Department.  TMSA was quite a specific case.

The thematic reports that ICAI has developed this year, for example those on how DFID learns and on the private sector, show what, for us, is a helpful development of the methodology of ICAI.  It looks more broadly across the Department.  In both of those areas, ICAI was looking at some really challenging sets of issues for DFID.  I remember talking to the commissioners in advance.  Particularly in a Department as complex as DFID, dealing with issues and countries with the complexity that we do, there are not many easy answers on organisational learning.  We took the report on learning, even with an amber/red rating, as a really helpful input into our own development in this area, and similarly on the private sector.

We do not see the pattern of traffic lights this year as concerning in itself.  We welcome the way that the ICAI methodology and scrutiny is developing. 

Q47   Chair: Do you accept that both the private-sector and the learning reports grew out of the fact that, having looked across their reports, ICAI concluded that you had a problem in those areas?  Effectively, whilst lots of people knew lots of things, your institutional learning system was not effective.  On the private sector, people have often said that the DNA of DFID is not naturally affiliated to that.  Whilst they were able to identify one or two specific things that you have done quite well, they implied that, culturally, you were not yet comfortable.  Do you accept that?  Does that have a real impact on what you have to do?

Richard Calvert: I will take the private-sector one first, and let Nick say something on learning.  On the private sector, clearly we have been challenged, not least by our Secretary of State over the last 18 months or so, to really step up our work with the private sector. 

Chair: We have challenged her as well.

Richard Calvert: It is a very fair analysis of DFID to say that, if you look back over the last five to 10 years, we have not have a strong private sector DNA in the organisation.  We are working very hard to build up our work in this area.  We have brought in a lot of new skills.  We are developing what we feel is much more coherent programming in this area.  We have set out a high-level vision.  While the ICAI study clearly identified a number of areas where ICAI felt we were not as strong as we should be, a lot of which we would acknowledge, we would also say that we have been moving quite fast over the last six to 12 months, and we feel we are in a stronger position now than when the work was undertaken.

We will be very interested to see the follow-up work that ICAI is doing on the private sector.  Obviously, the first study focussed on three specific countries in our portfolio. There will be in three specific countries follow-up work as part of the Year 4 programme.  Certainly, we hope that ICAI will see some of the development the Department has made in the meantime. 

Q48   Chair: Are you able to say what particular lessons you feel you may have learnt during the past year, as a direct result of ICAI’s report, leaving aside TradeMark Southern Africa?

Richard Calvert: There was a challenge around coherence, and how coherent our portfolio was in this area.  If you look back at the development of DFID’s private-sector work probably over the last three or four years, it developed in what was probably a rather piecemeal way.  In some countries, we had quite strong portfolios developing.  In India, for example, we had quite a strong private-sector portfolio.  We had activity through a number of wider instruments, such as CDC, which is an important part of our work, and the Private Infrastructure Development Group, which is important. 

However, we had not got a sufficiently coherent view across the Department as a whole about our activity.  That is one of the things we have certainly taken from the ICAI report.  We are just in the process of completing our annual resource allocation process for 2015, 2016 and beyond.  That has brought together, for the first time, a really coherent view of the overall economic development and private-sector portfolio.  ICAI challenged us very reasonably on our capability and the guidance available for staff in this area.  Again, that is something we have made good progress on.

A lot of the things that ICAI picked out are ones we would have recognised we were making some progress on, but part of ICAI’s role is to really keep us up to the mark in making progress in these areas.  Perhaps Nick could say something on learning. 

Nick Dyer: The learning report came out at the same time as a review that was done by the Cabinet Office on what they call What Works, or how we use evidence within DFID.  We actually had two similar reviews at the same time.  The review told me that we have come quite a long way.  If you look at the use of evidence in DFID, in terms of the reports that come out of our Quality Assurance Unit, and if you look at the evidence survey we did of staff and their perceptions about taking on evidence more seriously, we have come quite a long way.  In fact, the Cabinet Office says that we are a leader across Whitehall in the use of evidence.  Now, that may say more about the rest of Whitehall than it does about DFID, but they were very complimentary about it.

At the same time, the message is coming out that it is not as consistent across the project cycle or across the office.  These are messages that we would recognise.  We have, over the last couple of years, tried to improve the generation of our evidence to transmit it better.  We have made various system changes to try to get the evidence working and moving more quickly and for people to adopt the evidence.  Clearly, we have got more to do.  We are actively following up the review by doing two things: very clearly building in messages around using evidence into our new Smart Rules that we are developing for the project cycle, and setting up a task group who are going to take the What Works review and the ICAI review, and identify practical actions in terms of moving forward. 

Q49   Sir Peter Luff: Do you see ICAI more as an honest friend than a threatening foe? 

Richard Calvert: Yes.  I do not think we see it as threatening.  Any organisation doing the kinds of things we are doing, which are difficult things in difficult places, should not feel that it has not got things to learn from scrutiny and challenge.  Clearly, it is important for us that the relationship with ICAI is based on openness and honesty both ways, and that the processes around ICAI reports are reasonably proportionate and efficient.  With those qualifications, we feel that ICAI is there to help and support us deliver better value.

Q50   Sir Peter Luff: On the question of processes around reports, you produce a management response to each of their reports.

Richard Calvert: Yes.

Q51   Sir Peter Luff: ICAI wants them to be more consistent.  They say that when you accept improvements are necessary and state that you are doing some of the necessary work already, you should be more fulsome in accepting the recommendations—so accept the whole recommendation—and list the full management actions and fill in in detail how you will respond to their recommendations.  What do you say to that?

Richard Calvert: Overall, our response to ICAI recommendations is very positive.  We have accepted the vast majority of recommendations.  We try to set out in our management responses a reasonably clear and succinct statement of what we are going to do to follow those up.

We are always open to further discussion with ICAI about the detail of particular reports or particular follow-up action, if anyone feels that is missing.  ICAI obviously take seriously its follow-up work on previous reports, which gives a chance to check follow-up action.  Overall, we feel that the processes at the moment are generally proportionate and clear.

Q52   Sir Peter Luff: Do you think you could be more consistent?

Richard Calvert: There has clearly been a challenge, particularly around whether we have been consistent in saying whether we accept or partially accept recommendations.  In fact, only yesterday we talked to the ICAI commissioners about that.  Firstly, we look at each report on its merits, and we look at each set of recommendations.  Ministers will look at those as well, and decide exactly how they want to respond.  It is fair to say that, over the last year, there have been a number of areas where ICAI had made recommendations on issues that we felt we had already made quite good progress on. 

This is reflected to some extent by what we were talking about earlier with, for example, the private-sector work, or some of the other areas where things are moving quite quickly in the Department.  In those cases, we have sometimes said, “We partially accept your report, because actually we feel we have done quite a lot of it,” rather than simply accepting it. 

We accept that we could perhaps be a little bit more consistent in some of those areas, but we do not want to put consistency above the appropriateness and relevance of response, case by case.  This does not feel to us like a major issue, providing the content of the management responses is serious.

Q53   Sir Peter Luff: In a sense, it is not really the detail of what is in the page and what you say in response the report; it is the spirit of the exchange between your organisation and theirs.  It is a sense of mutual shared understanding and objectives.  So, what else do you do, in your work with the rest of the Department, to reflect ICAI’s recommendations to the broader organisation?

Richard Calvert: It depends very much on the kind of report.  So, some ICAI reports are quite specific and quite focussed on particular countries and programmes.  Some of them are much broader.  I will take some examples of areas where we tried to pull out the wider learning.  We are doing a lot of work at the moment on strengthening our programme management procedures.  There are lots of things in different parts of the ICAI body of work that tell us things about ways we can strengthen our programme management.  Nick has mentioned some of the examples; we have tried to reflect those in the new set of rules and guidance that we have put out in the organisation on programme management.

In other areas, it is more about taking themes in a particular area like the private sector, and making sure that we are picking up key themes in the guidance we are putting out of the Department.  In some areas, ICAI’s recommendations specifically will have much more prominence.  For example, there are not many people in DFID who do not know the TMSA report and the detail of that.  That is fine; we are completely comfortable with that.

Q54   Sir Peter Luff: Is that informal channels?

Richard Calvert: That is in formal channels as well.  We have done a lot of formal communication on TMSA and the lessons that we needed to learn from that.  There, we gave that report, as an ICAI report, a lot of prominence.  In other areas, ICAI will perhaps sit behind some of the things that we are doing in the organisation.  Although it will be communicating with the organisation on learning, private sector and other things, it may be less framed around the ICAI report itself.

Q55   Sir Peter Luff: One of the hopes we all have for ICAI is that it will legitimise aid spending and make the investment more acceptable politically.  You got a pretty good billing from ICAI commissioners in the last session we have just had.  He said there are a lot of lights hiding under very large bushels at DFID.  You should be more confident and promote yourselves more assertively.  He said you are doing a really good job.  Do you sense that empowerment from ICAI’s work?

Richard Calvert: As a Department, it is incredibly important to us that we can tell a positive story about how we are spending money and what results we are delivering.  There are a number of sources of challenge and assurance that we use.  ICAI is a really important one.  The NAO scrutinises us.  We have got our own departmental results framework and reporting.  Although we are not in any way complacent about what we do, we do feel that there is a lot of positive endorsement of the Department’s work, in all those sources of assurance and scrutiny.  ICAI is one of those. 

Q56   Sir Peter Luff: So, you feel confident and empowered.

Richard Calvert: Yes, but not to the point of saying that we have not got things to learn.  Clearly, TMSA landed on us and was a major issue that we needed to deal with.  Given what we do and the places we work, we have no room for complacency.

Nick Dyer: ICAI is a good thing.  The organisation and the people who come across ICAI regard it as a good thing, and we have responded well to its recommendations.  In terms of using the feeling of empowerment in terms of how well we are doing, part of that is how we communicate our successes.  Cutting through, as we all know, is quite tricky.  I suspect the press are more ready to pick up on the negative stories than the positive stories.  We just have to work with that.  We are working hard as an organisation to try to tell the positive stories, in terms of how aid works and what value it brings to the UK.

Q57   Pauline Latham: ICAI said your corporate results agenda has brought greater rigour and discipline, but it can also distort programming choices.  For example, it says that pressure for early results has led to a focus on “quick wins” and easily scalable outputs.  Do you agree or not?

Richard Calvert: We do not agree on that.  When we first put in place the departmental results framework in early 2011, we were very conscious of the fact that much of what we do is not susceptible to a simple quantification of results.  The framework that we set up, and the resourcing plan that we put around that, very consciously did not attempt to turn every programme we do into a set of published results.  On the other hand, there are lots of things we do where we feel it is absolutely right that the Department can tell a clear story about what it is delivering.

We have been very conscious over the last three or four years of keeping this balance right.  We should be clear about results that we intend to deliver.  Where they are clearly measurable, we should be accountable for those and we should publish information.  Equally, there are lots of difficult things that we do, and will need to continue to do, that simply do not fall into that category.  Within the Department, we feel that the guidance we have had from Ministers and the decisions that have been taken on continuing to do difficult things in difficult places enable us to keep that balance about right.  Pauline or Nick may want to add to that.

Nick Dyer: The other part of the criticism is that we are choosing the easier results to measure, rather than the more difficult results, in terms of impact.  That is a wider challenge, in terms of whether we are going to measure our contribution or our attribution to results.  There is a question as to whether it is realistic as an organisation to be measuring real world outcomes that are a function of lots of other things happening, or if we are going to focus on the outputs or even the activities.  If you look at our results framework, we have got a range of different results across the different levels.  We probably need to be a bit more consistent on that.  Focusing on what is within our control is probably the fairer way of measuring if we are having success.

Q58   Pauline Latham: Look at something liked gender-based violence.  The Secretary of State has put women and girls right at the forefront, and there has been quite a lot of work done on FGM.  We witnessed that in Ethiopia.  Sensitive, evidence-based approaches can take a very long time for behaviour change when you are talking about things happening village to village.  A country like Ethiopia is huge, and you have got to take the population, the Imams and the faith leaders with you.  Because it is still a very male-dominated society, you have got to take the heads of the villages with you to change that behaviour.  In areas such as this, how can you balance that with having to demonstrate the results that you need to deliver?

Nick Dyer: That is a really good example of an issue that you really have to stick with.  It is going to take a generation; this does not happen overnight.

Pauline Latham: Yes, probably more than that.

Nick Dyer: Perhaps more than that.  We have got a 10-year programme in Sudan.  It has committed itself, albeit over two phases, because it recognises it is going to take a 10-year frame.  It is necessary to be realistic about measuring the interventions that we are doing and tracking whether they work or not, because this is a contested area.  You can fall back a little bit on indicators of social change and attitude change, which you can measure as you go along.  It would be unrealistic to expect the impacts, in terms of actually changing the practices, to happen in year one.  You have to get measures about whether your interventions are happening, whether they are working and whether there are changes to cultural perceptions—and you can measure that.  I agree with you that, in terms of actual impact, it would be unrealistic to expect that to happen from year one. 

Q59   Pauline Latham: It is very difficult to prove that you have changed anything, because the concrete examples are not necessarily there yet.

Nick Dyer: They are not.  One of the challenges that the Cabinet Office, in their What Works study, put to us was that we should do more quantitative trialling within DFID.  You could take an example of an intervention in female genital cutting, and have one intervention in one area and a different intervention in a second area, and compare and contrast as to whether they are having a different impact.  There are technical ways you can to do that to bring to life the “how”, in terms of what is working, rather than the “what”. 

We have taken that message away and have started to think about how we could do that.  I agree.  In areas where, frankly, we do not know what works, you have to test it as you go along and take some punts.

Q60   Fabian Hamilton: We have already discussed TradeMark Southern Africa and that now-famous ICAI red-rated report from December of last year.  Can I come back to that?  In your response to that ICAI report, the Secretary of State immediately closed TMSA down, and announced that programme controls would be strengthened throughout DFID and that improvements would be made to DFID’s internal audit.  I wonder if you could update us on what action has taken place since then.

Richard Calvert: The TMSA report coincided with some work that we were already starting to do in the Department on reviewing our programme management procedures with a view to strengthening the delivery as well as the design components and, in some cases, streamlining the way we worked. 

What has happened specifically since then is that we immediately had an exercise across the Department to ensure that we were more consistent in completing, and ensuring that sufficient attention was given to, annual reviews.  One of the lessons from TMSA was that the annual review process was not working as well as it should have done.  We had an immediate exercise in December and January to make sure that we were up to date across the Department with all our annual reviews.

Secondly, we have been working on a new set of programme management procedures and control points throughout the programme cycle.  This has been brought together in something we have called the new Smart Rules for the organisation, which you may have heard about in other sessions.  Smart Rules is, essentially, the new programme management tool kit for the organisation.  That was launched on 30 June.  A key element of that, which also picks up issues that were raised in TMSA, is the establishment of a senior responsible owner, for each project intervention in the organisation. 

One of the issues that I personally took from TMSA was lack of clarity about who was really in charge of managing the programme.  So, all programmes will have a senior responsible owner.  We have an initial cohort of 50 SROs who have been identified and who have had an initial training session.  We will be using that cohort, which represents all the business units across DFID, to roll out this SRO model across their teams and across all interventions.  By the end of 2014, we will have SROs in place for all our activities.  We have implemented a programme of programme management accreditation across the organisation. 

Again, one of the things we had recognised in the organisation was that the programme management profession was perhaps under-recognised, set against some of the other professional groups in the organisation.  So, we have created a framework for programme management professionalisation. We have had an initial group of people accredited as programme managers.  We have put in place a learning and development programme, for both programme management staff and those aspiring to programme management roles to undertake accredited learning.

Within the organisation, we see this set of issues as probably our biggest single internal change exercise over the 2014 to 2015 period.

Q61   Fabian Hamilton: Was most of that a reaction to that ICAI report?

Richard Calvert: We kicked off what we call the end-to-end review of programme management in early 2013.  So, that work was kicked off around February and March.  What TMSA did was give us a bit of a wake-up call on that.  If you like, it added some extra edge and challenge to that process.  We would have been on that track, but TMSA gave us some—

Q62   Fabian Hamilton: It was like a catalyst, to speed the whole thing up.

Richard Calvert: Yes.

Q63   Fabian Hamilton: I wondered whether, with your existing DFID instruments for, for example, supporting the private sector, there were ways to boost investment and infrastructure, such as the Private Infrastructure Development Group, and whether you are confident that there has not been damage to all those other bits, for example through the Private Infrastructure Development Group, that have worked successfully?  Infrastructure is so important in development.  Also, we are concerned on this Committee that, following what happened with TMSA, you might have become excessively risk averse.  Can you reassure this Committee that you are still prepared to take risks where the rewards are great, but that you are monitoring them adequately and properly so that we do not have another TMSA?

Richard Calvert: We very much believe that is where we are.  Even if you take the challenges that TMSA were seeking to address, to promote trade and development across Southern and Eastern Africa, that is not something that we are going to give up on.  We are looking at the moment at how best we can support that going forward.  There are clearly important lessons to take from TMSA about how we do that.

Equally, across our other instruments supporting investment or economic development, we see that portfolio continuing to grow.  Also, we feel there are some real successes in that portfolio.  I do not at all think that the issues with TMSA have left us feeling that we cannot go into that area of trade and development and private-sector development work.

Q64   Fabian Hamilton: So, you are not more risk averse than you were.

Richard Calvert: No.

Nick Dyer: Part of the change in the programme management rules is about shifting the culture of the organisation in two ways.  One is to focus less on design and more on implementation, which is a consistent challenge from ICAI and others.  The other is to empower people to take accountability for their decisions, so that rather than it all coming back to the centre, people can take decisions themselves.  Part of that needs to go alongside a culture of avoiding a risk of people being afraid to fail.  That is an issue, particularly in the world of scrutiny and criticism.  If you are in a space where people feel they are just going to get criticised and punished, then that is not going to work.

How then do we do that?  It is twofold.  One, it is giving the signal—and as a management team, we are very much clearly giving the signal—that well-managed failure is fine.  That is a slightly odd phrase.  What we mean by that is so long as you are following the systems and the rules, and you are doing your projects properly, things happen.  Things do not work.  If they do not work, that is fine; if they fail, that is fine.  So long as you manage your processes well, that is an okay outcome.  Badly managed failure is not fine and should not be tolerated.  That requires a shift in quite a lot of the culture in the organisation and for people to feel that they can raise issues and problems early.  We are certainly giving those signals, and certainly I hope that, as an organisation, we are starting to surface more things that have gone wrong and we are identifying them earlier.

Q65   Jeremy Lefroy: Given that the organisations with which DFID works would be expected to have proper financial controls in place, would it not be possible just to ask them for a very simple figure at the end of every year of funds they have received that have not yet been applied to the programmes that they are contracted to deliver?

Richard Calvert: That is something that we would expect a good programme manager to be doing.

Q66   Jeremy Lefroy: Could that not be compulsory across all programmes?  Then, something like TradeMark Southern Africa would just be flagged up immediately.

Richard Calvert: The clear expectation—and it is reflected in the guidance we have—is people should know what is happening to the money they are responsible for.  So, if that has been lodged with an intermediary for good reasons, they should be completely on top of that.  Clearly, in TMSA, we were not.  The problems with TMSA were not so much that the processes said it was okay but that we were not doing what we should have been doing.

Q67   Jeremy Lefroy: If I could just come back to that question, is it not possible just to have that figure, which enables you to say, “Yes, we have got a problem here, because of the £20 million that we gave them, £18 million is still there”? 

Nick Dyer: Clearly, we should not be paying in advance of need.

Jeremy Lefroy: But we do.

Nick Dyer: There have been instances in TMSA where that was one of the challenges.  There were two reports we should expect every single project to be doing.  One is its annual review, which is looking at the development impact, and the other is its financial report, which may go alongside an audit report as well.  The challenge for us as an organisation is that we actually take those reports, look at them and respond to them.  If you look at how many annual reports have been late and delayed in the past, we have now corrected that.  Clearly we were not doing that well enough.  That is an expectation of good programme management.  That is something that will be part of the training process.  A senior responsible officer, who is responsible for a project, would be expected to know that.

Richard Calvert: We also have clear rules in the organisation that payment in advance of need is not allowed.  Our auditors—the NAO, as well as our own internal auditors—look very closely at payment in advance of need every year.

Q68   Jeremy Lefroy: At the end of 2013, as far as I remember, we made some very substantial commitments to both the Global Fund and IDA, as part of our 2013 aid spending.  Were those given as cash or were they simply as commitments?

Nick Dyer: They were given as commitments.  We may well have lodged a promissory note—which is basically a Bank of England promise-to-pay—but the cash is only drawn down and released when it is needed.  In many of our multilaterals, that is how we operate.  They need the commitment, certainly in the Global Fund’s case, to be able to do the programming and make commitments themselves to countries.  However, they do not actually ask for money, and we do not give them money.  The Bank of England does not release the money until it is actually required.  That release schedule is based on their liquidity policy and an understanding about the cash flows of other donors, and then you make a decision about when to lodge the promissory note and when to draw it down.

Pauline Hayes: At the field end, staff go into a lot of detail before they agree to release funds.  They check in advance of disbursement very carefully.  They know that they are not supposed to advance in that way.  Occasionally, something will happen, because we obviously ask our partners to give us that confidence and assurance.  Sometimes, in some environments, stuff can happen and the money disburses more slowly, for example in Afghanistan or Pakistan.  Generally speaking, however, that rule is very clear to staff.

Q69   Jeremy Lefroy: Nick, you explained very clearly that it is quite possible to have well-organised and well-managed failure, which I do understand.  Can you give me examples of programmes where that has happened?  How do you report it?  The annual report tends to be full of stuff that is working, which is great and very positive, but are you open about this publicly, as well?  Do you say, “Yes, we have got programmes where everything was well-arranged and well-managed but, due to circumstances which are completely beyond our control, we have not been able to deliver those results?”

Nick Dyer: I will give two examples.  One is an example of what does not work.  We recently funded a trial for children who have severe malaria.  Past practice has been to rehydrate them—give them liquids.  We now know that is actually dangerous, and we should not be doing that.  We have stopped doing that; we have stopped doing that in the UK as well, on the back of this trial.  That is an example about understanding what works and implementing that message. 

On projects that are not working as well as we would expect, we do now, as a management team, get a regular report of all the projects that are scoring Bs and Cs, in terms of our reporting.  We interrogate those at the board, because they come through.  We are now flagging up and identifying early projects that have got issues and problems.  Some we have closed down, as well. 

Q70   Jeremy Lefroy: How long has that been going on for?

Richard Calvert: We put in place the new scoring process about three years ago.  The internal reporting has stepped up, as part of this work over the last year on programme management.  The consistency of our scrutiny at board level has changed over the last year. 

Q71   Chair: ICAI’s report questioned the sustainability of the programmes in Afghanistan.  The Committee has also highlighted that issue.  They recommended that you consult more widely with the Afghan people.  Is that happening?  How is it happening? 

Pauline Hayes: We took on board that recommendation, and we are endeavouring to consult more widely with the Afghan people.  We are designing projects and programmes as part of the normal design process.  You will appreciate that our ability to consult widely is quite constrained, in that environment.  To the extent we can, we are trying to react in a positive way to that recommendation.

Q72   Chair: Presumably, you can find some reliable Afghan partners?

Pauline Hayes: Yes.  We can do it through third partners and stuff like that.

Q73   Chair: How has the election changed things?  Do we know what the result is yet?

Pauline Hayes: The initial results were announced yesterday, but they are very preliminary results.  There is some way to go yet.  The situation is quite fluid, shall we say, at the moment in Kabul, because the losing candidate is obviously not accepting those preliminary results at this stage. 

Q74   Chair: Is Ashraf Ghani still supposedly the winning candidate?

Pauline Hayes: Yes, from the preliminary results.  There needs to be more auditing of results done.  The two main electoral commissions need to work together on this, to provide assurances of the results. 

In terms of how the elections went, the first round went extremely well.  It went very smoothly overall.  Obviously, we are now at the critical stage.  It is going to be a little bit difficult over next few days.

Q75   Chair: So, at the moment, you cannot judge your approach until you know the outcome.  Is that the situation?

Pauline Hayes: I do not think, depending on who wins, our programme and the shifts that we planned will change fundamentally.  We are aware of what both candidates have said about their vision for Afghanistan and their priorities, etc.  We are confident that we can work with either of them in the areas that we care about the most.

Q76   Chair: Is there likely to be a deal between the candidates, or is it going to be one winner and one disgruntled loser?

Pauline Hayes: I think it is too soon to say.  The situation is very much in flux, with the announcement yesterday. 

Q77   Chair: Okay.  I will not pressure you, in that case.  It is their politics, not ours.

Pauline Hayes: Exactly.

Q78   Chair: Obviously, the Committee felt very strongly about the need to prioritise gender issues.  You did adapt your programmes; indeed, you did change the emphasis as a result of that.  What progress do you feel you are making?  We made that the litmus test.  We said that the progress with women that is sustained over the next two or three years will be the test as to what extent our activities have made a real difference. 

Pauline Hayes: The big shift since you were out there is the support we are giving to women and girls, both from an economic development perspective but also a political perspective, in terms of their political empowerment.  We have been doing a lot of work to support female candidates in the run-up to these elections and the provincial elections.  That work will continue beyond this period, in terms of supporting them as they become elected representatives.  It is a capacity-building exercise. 

We have also upped our game in terms of access to justice for women, partnering up with Australia.  We are doing a lot more economic development work; we are actively engaging with women.  We have specific projects that benefit women.  Our Comprehensive Agriculture and Rural Development programme is designed to improve agricultural production.  We also have some private-sector work.  If you were to go back now, you would be very pleased in terms of what you are seeing with those changes.

The other thing you talked about was the importance of the agricultural sector.  We have just agreed a second phase of our Comprehensive Agriculture and Rural Development Facility, CARD-F.  We are giving great emphasis to that sector, and will continue to do so over the next five years.  Agriculture is obviously the bedrock of the economy there.

Q79   Chair: Again, somewhat echoing the Committee situation, ICAI found a risk that our aid to Pakistan would continue to increase, but against a background of a lack of concrete progress on areas like anti-corruption and tax reform.  As you know, the Committee took a very strong view that, unless the elite are making a proper contribution, it was wrong that the programme should escalate from ordinary taxpayers in the UK, effectively.  Is progress being achieved?  I know there has been a unit set up to work on the tax issue.  Are people actually beginning to pay tax, or even say that they are going to pay tax, given that they were not before?

Pauline Hayes: The programme has not scaled up as rapidly as we anticipated two or three years ago.  There are a number of reasons for that.  Last year Pakistan had elections, and that slowed down our work generally there, for some time.  It is also a recognition of the realities of working in Pakistan, and designing and developing programmes.  You will see that the trajectory has not been as rapid as we anticipated three years ago, but it is still going up slightly this year. 

In terms of tax, the new Pakistan Government, when it came into power last May, very quickly set to work on agreeing a new IMF programme.  That has a very strong emphasis on tax reform, as you are probably aware.  There are quarterly reviews of that IMF programme, the last one being last month.  So far, the Government has delivered on the targets, including on tax work.  They are shaping up well.  They produced a budget in June, which set out a lot of commitments that would assume a rise in tax.  The IMF has been looking at that quite closely, and will continue to look at it closely.  The acid test is now that they have set out their stall in this budget, can they deliver it over the next 12 months?

Q80   Chair: I do not know whether other members of the Committee did as well, but I got emails from concerned middle-income taxpayers in Pakistan saying, “We are worried that your report means that what will happen is they will just come after us.  They will want to get more out of us, and the people above us will continue not to pay.”  That seemed to be a perfectly legitimate complaint.  Are you satisfied that the people at the top, whether they are politicians or business leaders, will in fact make a contribution, given that a lot of them are paying nothing at all?

Pauline Hayes: I have heard a similar complaint when I have visited.  The phrase used was “milking the same cow”.  They definitely do need to get to the people at the top. A tax directory has issued MPs and others have to register their interests, and so on.  We have just got to keep the Government’s feet to the fire on this agenda.  We do raise it all the time at the highest levels.  So far, they have met their commitments and the actions in the IMF programme.  We just have to keep at it, because it is critical to get the tax take up. 

Q81   Chair: You have the Committee’s encouragement and support.

Pauline Hayes: I know.

Nick Dyer: One of the programmes that we have created is a capacity-building support from HMRC.  We are making that available to other countries on a demand basis.  One of the countries that HMRC is now working with is Pakistan.  There was an inward visit from the head of the Pakistan Revenue Authority.

Pauline Hayes: It has gone both ways.  We had the Pakistanis here, and HMRC have gone to Islamabad.  They have actually signed an MOU now, setting out the level of co-operation.

Q82   Fabian Hamilton: In its follow-up of its Year 2 report, ICAI was concerned at the lack of progress in the conflict pool and the significant risk issues that it identified in its report will continue to be magnified with the new CSSF.  I wonder if you could reassure us that that is not going to happen.

Pauline Hayes: Yes, certainly.  CSSF is a different beast.  It is definitely a different beast.  For one thing, it is not just three Government departments; all Government departments are taking part in it.  Also, every country has to develop a strategy that the NSC will scrutinise.  From that strategy, they will then produce a bid, which the CSSF will fund.  We are taking a more strategic approach to this agenda.  I recognise that ICAI was concerned that the monitoring and evaluation frameworks would be weak and so on, but we are setting up the mechanisms for the CSSF now.  A lot of work has been done in the last few months at the centre of Government on this.  Countries are producing their strategies.  The CSSF bids will be submitted over the next two to three months.

On the monitoring and evaluation issue, there still is a need to do more work there.  I would agree with ICAI on that.  We have an opportunity at DFID, because we are well used to developing these kinds of frameworks.  We can share what we have learnt with other Government departments, in terms of getting this right.  There is a new CSSF secretariat at the centre, and we have got a few DFID staff in there now.  Again, they can help shape the governance arrangements and so on for CSSF.  We see this as a different beast.  It is not just going to be more of the same.

Q83   Fabian Hamilton: Are you confident the Conflict, Stability and Security Fund, CSSF, which replaces the conflict pool, is not going to be involved with shoring up militarily some of the organisations and countries that apply for funding?  In other words, is there any danger at all that CSSF will be used for arms sales, or for anything that promotes violence?

Pauline Hayes: Well, of course not.

Q84   Fabian Hamilton: Forgive me, Pauline; it sounds like a strange question.  The reason I raise that is because I have had constituents who have expressed concern to me that this change from the conflict pool to the CSSF will actually allow arm sales to take place, in order to shore up stability.  In an effort to promote stability, it will involve armaments.  That is obviously a concern they have; that is why I am raising it. 

Pauline Hayes: I do not think that is going to happen, and we will certainly do our best to make sure of that.  I would say a large proportion of CSSF—I need to check the details—is classed as ODA.  That obviously ring-fences what can be done with it.

Q85   Fiona Bruce: ICAI remains concerned that DFID has yet to develop appropriate strategic guidance to cover the appointment of contractors and perhaps the learning process from them.  Could you tell us how you are going to address this?

Richard Calvert: We have got a big programme of work on strengthening our approach to commercial capability and the commercial performance of the organisation.  It is true that ICAI thought we should have some kind of strategic guidance on the types of contractors that should be used in different situations.  We have not taken the work forward quite in that way but, as the follow-up report recognised, we have done a lot to strengthen our capability and expertise across the organisation.  We have also put in place much stronger practical guidance for people who are taking decisions on programme design. 

We have taken it down a level in the organisation.  Clearly, the choice of individual contractors and the commercial approach needs to reflect the circumstances of each intervention.  From my perspective, we have followed the spirit of ICAI’s recommendation, in terms of saying, “You need more guidance and capability within the organisation,” but we have taken it forward in a slightly different way from their original recommendation. 

Q86   Fiona Bruce: That is interesting, because they say that when the choice is left to programme teams on a case-by-case basis, there can be a serious gap in the strategic approach to using contractors.  Are you addressing that?

Richard Calvert: I would have accepted that 18 months or two years ago.  I would not accept that now.  The things that we have done to strengthen our performance are: firstly, all our Senior Civil Service staff are going through a three-day commercial learning programme.  We have put in place a network of specialist commercial advisers across the organisation.  We have put in place a key supplier management process with our top dozen or so suppliers.  We have put in place much more lesson-learning and sharing, both within the organisation and between DFID and its suppliers.  Although I think some of those criticisms would have been fair a couple of years ago, now we feel in a much stronger position on that.

Q87   Fiona Bruce: Can you say why the amount of aid delivered by DFID through contractors has risen?  It has risen from about £0.9 million in the previous year to £1.4 million in 2013-14.

Richard Calvert: A lot of this reflects the growth in the programme over the last three to four years.  We have seen the DFID portfolio, as a whole, rise from a little over £7 billion in 2010-11 to in excess of £10 billion.  We have seen some shifts in the balance of work within the portfolio, but the growth in the contractor component is just reflecting that overall programme growth. 

Q88   Fiona Bruce: Were you in the earlier session?  Did you hear the evidence on the relationship with UNICEF?

Richard Calvert: No, I did not, but we know the case. 

Q89   Fiona Bruce: There was quite concerning evidence there.  We would very much want to know how you have developed your strategic oversight of multinationals and NGOs, to avoid that kind of issue reoccurring?

Nick Dyer: Our relationship with multilaterals is twofold: we give core support, through our replenishments, and we use multilaterals as a delivery mechanism from country offices.  They put money through multilaterals, in terms of delivering, as in the case of Nigeria and UNICEF that you were mentioning.  The expectation is that the country offices themselves are responsible for the delivery of the projects that they are agreeing with the individual multilateral.

In the core support, we have a range of mechanisms by which we are engaging with the multilaterals to try to improve their performance.  I know one of the challenges has been that we are not doing enough to bring our in-country experience to bear in our engagement with multilaterals.  It is getting better.  I suspect one of the challenges is that, even if we were to do that, across all our programmes we have still only got 28 country programmes, and many of these organisations are dealing in 70 or 100 different countries.  There, it is a question of what knowledge we have, alongside building and drawing on the knowledge of other donors and the multilateral systems themselves.

The example of UNICEF is quite interesting, because, in response to the ICAI report on UNICEF, we started doing these portfolio development reviews.  This was to learn the generic lessons from our work in-country, and take that back in terms of trying to shape our engagement in the core support that we give to UNICEF in New York.

Q90   Fiona Bruce: Are you satisfied that that kind of fairly major concern that was raised by ICAI will not be raised in the future?

Nick Dyer: In terms of the specific challenges in Nigeria, northern Nigeria—as you have seen from the press—desperately needs support, particularly girls in terms of education.  It is an incredibly difficult place to work.  The process since the ICAI report has been that we agreed to do a review after six months, through the annual review.  The annual review was done.  The project was then put on a performance improvement plan.  We have now got the second annual review just going through our internal quality assurance.  Once that internal quality assurance has been completed, we will make a decision about what is next in terms of the programme.

It is a question about whether these are specific issues for UNICEF in the very difficult circumstances of working in Nigeria, or they are more generic for the organisation.  We can take the generic lessons back to New York, but the specific challenges of working in Nigeria are for the Nigeria country team.

Q91   Fiona Bruce: What interested us was that there was another NGO working in Nigeria on a very similar project, which seemed to be more effective.  Is there not a concern that you have these very large multinationals that have been established a long time, and that there is not sufficient scrutiny by DFID, on a case-by-case basis, of what they are doing?

Nick Dyer: That raises two questions; one is the scrutiny.  I would argue that the combination of our in-country, project-by-project scrutiny plus what we are doing at board level actually does amount to a sufficient amount of scrutiny.  The second issue that is implied by your question is whether we are giving enough competition to the choices we make on whether we put our money through UNICEF rather than through an NGO.

Fiona Bruce: Absolutely.  That is another question, indeed. 

Nick Dyer: I would say three things to that.  In many of the places we work, there is not a huge amount of competition because these are very difficult places.  Where the competition might exist, it cannot work at the scale we want it to, but that is a question of us engaging with that.  The third element points back to what Richard said on encouraging our staff to take a more commercial, market-shaping approach to our work.  The work that we have done on commercial advisers and commercial skills will make and is making our staff more cognisant of the fact that, actually, part of our responsibility is to shape the market as well as to direct resources through one organisation.  I would hope that we will start getting better at that, because you are absolutely right; we should not just accept that UNICEF are the only option.  We should be looking at the wider market and making choices that will build that market as well as deliver the project.

Q92   Fiona Bruce: You should not accept, because UNICEF has a certain reputation, that the scrutiny or the oversight does not need to be as strong as with any other project or any other organisation.

Nick Dyer: Organisations evolve, and we have to keep our scrutiny up.

Q93   Chair: ICAI were also concerned that you seemed to be taking too long.  They understood that you would have finished this process by the end of last month, and you are now telling them it will be a few more months. 

Nick Dyer: I checked this morning, after we had spoken to the commissioners yesterday.  The annual review has been completed, but because of the seriousness with which we are taking this, we are putting it through a peer review, internally.  We are putting it through our quality assurance unit just to make sure that the methodologies, conclusions and analysis were up to scratch.  That process will be finished some time this month. 

Q94   Fabian Hamilton: You have recently decentralised your evaluation function.  Are you comfortable that there are clearly defined boundaries between your self-evaluation function, the work of ICAI and the NAO?  I wondered what you feel is ICAI’s value added on evaluation, given that their budget is so much smaller than your own self-evaluation function.  I know that you do not keep a total record of what you spend on self-evaluation, but ICAI estimates it is over £200 million.  That is huge compared with the budget of ICAI, which is about £3 million.  In its recent report on how DFID learns, ICAI said that DFID staff failed to apply lessons from your self-evaluation to programmes.  How would you respond to that?  Do you agree that information from evaluations should be made into a usable form? 

Nick Dyer: In terms of the relative function of ICAI, in respect of all the other scrutiny, internal and external, the triennial review looked at that.  The triennial review concluded that, particularly with the change in ICAI’s function to do more thematic studies in more depth but fewer of them, it was a good added value to the process of the scrutiny.  That is providing an independent view on some themes.  The learning study and the private sector study show the value of doing that.  Yes, there is still a good role for ICAI in relation to that.

In terms of the decentralised evaluations, in 2011 we were doing 12 decentralised evaluations a year.  We are now doing 40.  So, we have scaled up the number of evaluations.  In terms of how much we spend on evaluation, we are doing two types of expenditure on evaluation.  We are spending on our own programmes, and calculating the cost of running our own evaluation.  The evaluation framework agreement, which you discussed last year, to date has now agreed 44 contracts.  There are another 12 in process.  The total spend will be about £48 million.  I suspect the figure looks bigger because we also fund evaluations of other donors.  We fund some World Bank evaluations, and we fund other organisations who do evaluations to allow them to do evaluations. 

We are doing an awful lot of evaluation.  The challenge for all this evaluation work is whether we are doing the right evaluations and evaluating the right things.  We did a review this year of our evaluation strategy.  We have come to the conclusion that we are not necessarily evaluating the right things.  You want to evaluate where there are gaps; you want to evaluate where we do not have sufficient evidence to give us good material; you want to evaluate programmes of a particular size.  We want to become smarter at deciding which projects to evaluate.

In terms of lessons, last year we did an annual review of our evaluations.  We tried to pull all the lessons into one place.  We are going to do another annual review this year.  By September, we will have tried to pull together all the lessons from the evaluations that we have done internally. 

Q95   Fabian Hamilton: Nonetheless, do you think you can justify more than £200 million being spent?  That is a phenomenal amount of money. 

Nick Dyer: Go back to: do we know what works in development?  The answer is that, actually, in some cases, yes.  In a lot of cases, no.  There are huge evidence gaps in evaluation.  This is not just about DFID; this is about development.  Do we want to fill those gaps through our research, our evaluations and our own internal operations?  Yes, we do.  The return from knowing what works and stopping what does not work well justifies the investments we are making in evaluation and evidence generation more broadly.

Q96   Fabian Hamilton: Are you confident that the public, who fund this, will agree with you?

Nick Dyer: If we are demonstrating what we are stopping doing instead.  The example I gave earlier, on the FEAST trial, is incredibly good value for money.  We are stopping children dying, because of a bit of research and a trial we did.  We can demonstrate a line of sight between the research we funded and changes in practice that will save lives. 

Q97   Fabian Hamilton: Can I finally pick up on a question I asked the commissioners earlier?  I do not know if you were in the room for this.  I asked them about the transition to ICAI’s next phase, in May 2015, and whether it would be smooth and well-managed.  The response was, “Not as smooth as we would like, given the number of changes that are taking place,” because all of the changes are at the same time.  I wondered whether you were concerned about the lack of continuity regarding commissioners and the staff.  What would you like to see done differently in ICAI’s next phase, after May 2015?

Richard Calvert: We are happy with the overall plans for transition.  We recognise that there is a lot changing in spring 2015, but we recognise that some time ago we put in place plans, in relation to commissioners, contractors and the secretariat function, that we think will enable us to have a sufficiently smooth transition.  We are building in, for both the contractors and the commissioners, a degree of overlap. 

At the moment, as you know, we are in the process of seeking to recruit a new chief commissioner designate, and also a further commissioner who will be appointed over the coming few months.  There will then be an opportunity for them to shadow the current commissioner team through the first half of 2015.  We have got a process already under way to appoint a new contractor, and we would aim to have the new contractor appointed by around the turn of the year, 2014 into January 2015.  Again, that should give us time for overlap.  We have also agreed short extensions to the contracts for the current commissioners.  Also, we are prepared to look at the wind-down of the current contractor programme of work.

We are also talking with the commissioners and the secretariat about potentially shifting some functions out of the contractor responsibility into the secretariat team, to give a little bit stronger permanent capacity in the secretariat.  This is a process we need to manage, and we need to make sure we stick to the deadlines.  We are confident that we have got a process that can avoid a sense of everything stopping and then having to be restarted again in the spring.

In terms of where ICAI goes in its next term, to some extent that is clearly down to the new commissioner team: to shape the programme and to decide where they want to focus.  If you go back to the triennial review, that gives the main steer on where we see ICAI going in the future.  Clearly, that pointed towards, perhaps, fewer and more in-depth studies.  We are already starting to see some of that reflected in the way the ICAI work programme is going.  The core function, however, as Nick said in his earlier answer, has been really reinforced.

Q98   Chair: Do you have any problems with the possibility of ICAI relocating into DFID’s offices? 

Richard Calvert: There is a very good value-for-money argument for doing so, in that if we can accommodate ICAI without cost in the 22 Whitehall building, rather than paying a rent somewhere else in central London, that is a very good reason.  We have talked to the commissioners about this, and we are clear that if we do go down that road, which is a very serious possibility, then we will look to find them a sufficiently discrete corner of the building so that they can feel they have got their own space and independence.

Q99   Fabian Hamilton: You could build a Chinese wall.

Richard Calvert: Maybe even more than a Chinese wall—maybe a large, 18th Century wall.

Chair: Thank you very much indeed.  The relationship will continue, obviously, and I guess some of these issues will come up when we do have the Permanent Secretary in front of us for the annual report.  They certainly informed that relationship.  Thank you very much for today’s evidence. 

              Oral evidence: ICAI Annual Report, 2013-14, HC 523                            21