International Development Sub-Committee on the Work of the Independent Commission for Aid Impact
Oral evidence: ICAI’s review of DFID’s approach to value for money, HC 951
Wednesday 2 May 2018
Ordered by the House of Commons to be published on Wednesday 2 May 2018.
Members present: Paul Scully (Chair); Mr Ivan Lewis; Stephen Twigg.
Questions 1 - 39
Witnesses
I: Tina Fahm, Lead Commissioner, Independent Commission for Aid Impact; Marcus Cox, Team Leader, ICAI; Joy Hutcheon, Director-General for Corporate Performance, Department for International Development; Vel Gnanendran, Director of Finance and Delivery, DFID.
Witnesses: Tina Fahm, Marcus Cox, Joy Hutcheon and Vel Gnanendran.
Q1 Chair: Thank you very much for coming, Vel, Joy, Tina, and Marcus. Thank you so much for your time this morning. What we will do is just crack on. Tina, I know it is your first time with this one-panel structure, which seems to have worked well over the last few meetings; it makes for more of a dynamic discussion. Hopefully this will be equally as productive. I will kick off, and you can say anything you want about the report briefly at the beginning in your opening remarks. If I can just start with you, Tina, and look at the methodology behind the report, the review involved case studies of four countries: Pakistan, Nigeria, Uganda, and Malawi. On what basis did you choose those countries?
Tina Fahm: Thank you for the opening question, Chair. Given the nature of this review, looking at value for money, which is both a process and an outcome encompassing the entire DFID portfolio, we really wanted to take a wide view of all aspects of DFID’s engagement. The choice of countries, as you have stated, included ones from both Asia and Africa. We were keen to ensure that we looked at both fragile and non-fragile contexts, and the humanitarian and development contexts as well. Size and national context was also taken into account.
If I may just go through the countries briefly, with Pakistan there was a very large programme. Pakistan is a fragile state in which there had been considerable humanitarian assistance. Nigeria is a large country in west Africa with a large programme. There, again, there was diversity of programing. We had an opportunity to look at DFID’s work on tackling corruption and greater transparency. Uganda is a mid-size country in east Africa. It is facing significant development challenges as well as significant growth over recent times. Malawi, in southern Africa, is the smallest of the four. This is a country that had experienced considerable development challenges and is a major recipient of funds from DFID. Those countries in totality supported our desk reviews and helped us to triangulate our findings.
Q2 Chair: Thank you very much. You looked at 24 DFID programmes as part of the sample, including both centrally managed programmes and programmes in the case study countries. How did you select those samples?
Tina Fahm: We selected the samples to ensure that all four countries were represented. We took six programmes from each of the four countries. We ensured, as you said, that there were centrally managed programmes amongst our selection to ensure scale within the overall samples. Of the 24, we also ensured that within our scope we were able to review programmes that had not been as successful, as well as those that had performed well.
Q3 Stephen Twigg: Can I take us to the issue of the complex environments in which DFID is operating, and ask our witnesses from DFID first? As you will know, ICAI made recommendations about how you measure value for money in complex programmes. My understanding is that while you had a broad agreement to the principle, you were not keen to pursue quantified scoring against target outcomes as opposed to outputs. Can you tell us more about why you have reached that view?
Joy Hutcheon: Yes. Perhaps I could just start by saying how much DFID welcomes the review. It is a hugely important area; this is fundamental to how we achieve impact. It was good to see ICAI’s assessment of the effort that we put into this, the progress that we have made, that we are a global champion, and that we have integrated value for money into our programme management. We completely acknowledge that there are areas where there is more to be done, and we have a very high level of agreement with ICAI on what those areas are. There are very good partnerships since the work in 2015 on driving this area of value for money forward, and we very much intend to build on this report to do even more.
On the question of complex programmes and outputs to outcomes, there are some programmes we run where it is quite clear what the outputs are going to achieve: if you vaccinate a child, the evidence is clear that you have protected that child. There are other programmes where the evidence is less clear in terms of how the output is going to deliver the outcome. Our approach in those programmes is to require the programme to have a very explicit theory of change, which sets out why the programme thinks the output will deliver the outcome, and then, really importantly, what the evidence is for that and how strong it is.
We have, partly as a result of discussions on this review, just strengthened the questioning around the theory of change in the annual review, to ask the programme team to reassess it and to specify whether there have been any changes in that theory of change and in the evidence for it during the course of the programme. We have considered whether we should ask programme teams to score that. The reason we have decided not to do that is that when you score outputs, you are looking at data and you can be quite objective about it; when you score your judgment of the theory of change, it is more subjective.
You rightly challenge us on the objectivity of our output scoring, and we work very hard on that; we are very keen to maintain the robustness. We were worried about subjectivity. We were worried about having two scores when we have a long-established data series on outputs. We think this is something that we should be asking teams to think very deeply about, but give us a narrative answer on in the annual review.
Q4 Stephen Twigg: Can I ask Tina or Marcus how you respond to that? What is your view of that response?
Tina Fahm: We certainly welcome DFID’s response, because the initial response to our recommendation was very vague, and in some ways we were not sure of what the commitment was. We acknowledge that, particularly with the annual review framework, improvements have been made to that since completion of our work for this review. In fact, there will be an opportunity for us to follow up further next year when we do the follow-up. Certainly, we welcome the direction of travel as stated by DFID.
That said, I want to be absolutely clear that scoring simply on outputs is not good enough. We need to go further. This is about value for money, capturing value, and being clear about what is being measured. Also in our recommendation, we want to see the quality assurance around that assessment that ensures confidence and assurance from others that value for money is being achieved. Marcus, would you like to add anything further?
Marcus Cox: This is one of the key weaknesses in DFID’s control systems that came out of our report. It is really a combination of the review scores that are based purely upon outputs against targets, and a pattern of the targets being regularly revised over the life of the programme. As you have seen from the report, there is this clear tendency for programmes to start off with quite ambitious targets and then adjust downwards over the life of the programme.
One of the challenges that raises for us in looking at the programme score as a tool for ensuring value for money across the portfolio is that it is very difficult, when you look at those scores, to assess whether you are really looking at the performance of the programme, or the level of ambition of the targets in a world where those targets are, often for legitimate reasons, moved over the life of the programme. This was really, from our side, a double-barrelled recommendation: we need both to have a more solid basis for scoring the programmes, based upon the real-world results that ultimately it is looking to achieve, plus a set of controls over the setting of targets to make sure that those are set at an appropriate level.
Q5 Stephen Twigg: Joy, one thing that struck me when you were answering the question is that DFID puts conditions on funding for a lot of multilateral organisations now that are about outcomes. A lot of what you were saying in response to the ICAI recommendations is exactly what the multilaterals say to me about the difficulty of the conditions attached to DFID’s funding. Would you like to comment on that?
Joy Hutcheon: We completely agree with ICAI that focusing on how outputs are going to deliver outcomes is fundamental. We would not say that you can ensure value for money through an output score, and we are trying to find the right way to ensure that programme teams are focusing more and more clearly on that. We do have programmes that have been closed where they were achieving all of their outputs, but the team had tested the theory of change and discovered that the outputs were not delivering on the outcomes. In fact, one programme was over-achieving its outputs and not delivering its outcomes, and that programme was closed down.
We are concerned about going down the scoring route for the reasons I have given, but we absolutely agree that this is enormously important, and the review that we have just done of the annual review process has focused on this as one of the things that we want teams to put more emphasis on.
Q6 Mr Lewis: My first question is to Joy and Vel. In addition to the discrete programmes, all DFID offices are encouraged to pursue cross-cutting objectives: building national capacity, tackling corruption and so on. Measuring value for money in that sense is even more complicated. How do you go about that?
Joy Hutcheon: This is something that we have thought very carefully about in response to the report. In the last round that countries planned their portfolio, they conducted a poverty reduction diagnostic, looked at the barriers to poverty in the countries they were working in, and made judgments about which barriers were susceptible to intervention, and particularly intervention by us. In many countries this has led them to a set of things that they were going to want to achieve across the programme.
In Nepal, for example, the office has set three-cross-cutting objectives that they want to ensure that every programme is either contributing to or not undermining. At the moment, they are tracking those qualitatively, and they are looking at whether they can track them quantitatively. They are at quite high level and are about the relationship between spheres of government in Nepal and capability of government. They are quite difficult to do, but they are thinking hard about that.
It is fair to say that in the last couple of years we have had so much emphasis on aggregating results targets at a very high level because of the numbers of results targets, for example, in the 2015 manifesto. This portfolio tracking is something that we have put less emphasis on in the last couple of years. As we go on to planning for the spending round that has been announced for 2019, we will want to think again about portfolio management, and how we are identifying and tracking issues at that portfolio level as well as programme level and whole-department level.
Vel Gnanendran: Can I just add one point? This Committee has rightly challenged us on our whole approach to results. One of the other things we are doing is refreshing our results framework, and that process is underway at the moment. One of the outcomes we expect from that will be a move back to country-level results frameworks. They will perhaps not be as comprehensive as they used to be, but there will be a fit-for-purpose, country-level results framework. We are looking at how to measure cross-cutting objectives at country level.
Q7 Mr Lewis: Yes. Our experience of targets in Government—and I am sure it is the same—is that you ask people to join up, and then you give them different performance measurements and accountabilities. People behave logically, and so therefore they chase what they are being judged on, and if they are being judged on specific things that may be contradictory or not aligned, then that leads to all sort of dysfunction. I might be wrong, but I would imagine that quite a high proportion of DFID’s in-country objectives require a joined-up approach, not a siloed approach.
Joy Hutcheon: When I looked at the Nepal work, I thought they had the formulation of this quite right, actually. They identified the programmes that needed to actively contribute. There may well be other programmes in the portfolio that are not key to delivering that objective, but you do need to make sure that they do not get in the way or do something that is undermining the objective that you are trying to achieve.
Vel Gnanendran: On your points about targets, Mr Lewis, one of the lessons from our consultation on the results so far has been that setting quantitative targets has not always been helpful, for the reasons you have said. The direction we are thinking about taking would be tracking results but not having them as targets. We would be seeing how many children are being educated in a particular country, but not setting a target, which is what we used to do before. We would be able to track that more explicitly across a set number of indicators.
Q8 Mr Lewis: Progress and completion is probably also incredibly important, not just the number of kids accessing school, as an example. That would vary depending on the stage you were at in a different country. In some countries, just simply getting kids into school is a pretty big achievement. In others, actually we need to move very quickly to progression and attainment. That is really important. Tina and Marcus, how do you feel about DFID’s response to this specific issue and the framework that they are seeking to put in place in terms of that cross-cutting work?
Tina Fahm: Thank you for asking that question, because I happen to think it is very important within the context of value for money. From our review, we are content that, in terms of baseline value for money, DFID has its ducks in a row, so to speak. When we are talking of cross-cutting, we are now looking at a different level of ambition. It is not just the basic accountability; it is looking at how the sum of the parts is much greater. It is getting the absolute maximum for every pound of taxpayers’ money. We know that, for example, WASH programmes are great, but if you add disability and ensure that beneficiaries can really access these services and investments, it is going to result in value for money for ODA spend.
In terms of DFID’s work on resilience, resilience is something that encapsulates so many different areas about building that capacity, and it takes time to build the collective. Also, building national Governments’ capacity to engage more in terms of delivering for their own populations may involve perhaps commissioning locally: local partners delivering parts of programmatic work. On paper, that may seem more expensive, but if that is going to deliver for a longer period of time for beneficiaries, then that has to be good news for the aid budget. This whole cross-cutting approach—and commitment to it; not just a stop-start—is very important and a key theme from our report.
Marcus Cox: That is exactly right. To elaborate on that, one of the ways we are thinking about the value for money question at this larger strategic level is around the question of trade-offs. If you are going to push these cross-cutting objectives, it is going to have implications for how you think about value for money within individual programmes. If you are going to build water points with access for disabled people, it is going to have implications for your unit cost and so forth. There are trade-offs inherent in trying to achieve value for money, and those things need to be better articulated at the country portfolio level, so that you can then apply them effectively in individual programmes.
Q9 Mr Lewis: Something that is more expensive may actually provide better value for money, because it is getting to harder-to-reach groups and achieving sustainable change. The risk when we talk about value for money is that you look at the bottom line and that is the end of it. Do you think DFID has an issue with that?
Joy Hutcheon: I do not think the issue is landing with country teams. We are not asking them to find the biggest number of results for the lowest cost. Our guidance on value for money is very clear on that. Our equity guidance is super-clear on that. The number of programmes that ICAI found that have factored reaching marginalised groups into their value for money assessment tells that story. There is a programme in Pakistan where building national capability was included, rather than a slightly higher technical economic return of delivering more results.
Teams have absolutely got that. The question is how explicitly we are articulating those objectives and whether we are tracking them or not. The reason we partially accepted this is that we are not absolutely sure we want to characterise all these as VfM objectives. We think these are impact objectives. They are a bit broader than VfM and they will enable us to deliver VfM through the whole programme.
Q10 Mr Lewis: I have a final, very quick question. Have Ministers discussed this with you in the context of the ICAI report? Clearly, if there is not political buy-in to what value for money means, that is going to lead to all sorts of tensions.
Joy Hutcheon: This question of trade-offs comes up all the time with Ministers. Clearly, as we go into the planning for the spending review, that will become very explicit. Ministers will make choices about what objectives they want to achieve where. We are not in a position to pre‑empt that, but it would be surprising if marginalised groups did not continue to be a cross-cutting objective for our delivery, because they are so fundamental to finishing the job on poverty reduction.
Q11 Chair: I just want to come back to the targets that we have talked about a little bit. Marcus, you mentioned changing targets earlier on, about the fact that the report talked about, whereas there may be good reasons to change targets midstream, the sense that they are not always explained as clearly as they might be. There is the suggestion that, in some cases, it might be setting unrealistic targets to then reduce the ambition later. What do you say to that view?
Vel Gnanendran: Again, this is an issue that we recognise. Our internal quality assurance unit also identified it; it spot-checks our annual reviews each year, so it is right to raise this. There are obviously very good reasons why we change targets. In my previous role, head of Tanzania, one of the most common reasons was optimism bias in the design phase. Quite often we underestimate how long the procurement takes. Often the first year of a programme is just procuring the supplier. Also, context changes. The politics of your country may change, and so the Minister who was championing your programme in a particular country may not be there. They get moved out and then the programme loses momentum.
There are lots of very good reasons why targets change. We have a baseline rule that any significant change to targets has to be approved by the head of department, but we have accepted this challenge from ICAI. With the change to our annual review template that happened in April, we have now put in new sections that ask teams to explain any changes they have made to the log frame and any future changes they plan to make. Also, the log frame tool will now track changes, so there will be an audit trail. We did not used to have visibility of previous changes. Those used to just be made and not recorded. We have now amended the system to allow us to see that.
Q12 Chair: In terms of the optimism bias example, how would you then move back to the original economic base that you made the decision on? How would you move it forward? What would you do to go right back to the beginning and say, “We have set out the targets on that basis. Things have changed, but we need to…”?
Vel Gnanendran: Again, that is a question that ICAI raised in its report. At the moment, the head of department will make a judgment as to whether those target changes have been so significant that the team should revisit the economic appraisal at the beginning. Our internal unit has said we probably need to put a bit more system and governance around that. In the new annual review template, responding partly to ICAI and partly to our internal assurance unit, we have put in a new page in the value for money section. In that, teams have to set out whether there is any reason, in their judgment, to go back to the original business case and reassess the economic appraisal.
One of the reasons we only partially accepted this recommendation is that reassessing the economic case is a big job. If we specify that up front, which is what the ICAI recommendation said, the risk is that we might get the timing wrong. It might be that the changes have been relatively minor, but then we have specified a point at which we have to go back to the economic case and we have to do all that work for relatively minor changes. On the other side, there might be a quite significant change to the context, but, if we have specified the review point as a year later, the team might just delay reviewing the economic appraisal for another year.
The reason we only partially accepted this part of the recommendation is that we did not want to specify up front in the delivery plan when teams should go and revisit the economic case. We wanted to strengthen the challenge they put on themselves to ask that question a bit more systematically as they go through the annual reviews.
Joy Hutcheon: In every annual review we are now asking the team to assess the value for money of the programme as it stands compared to the business case. As Vel says, they would need to make a judgment about whether to re-run the full economic appraisal to do that. We then ask them to say whether and why the programme should now continue, on value for money grounds. Again, that requires them to take a fresh look, not just rely on something that was done when the programme was designed.
Q13 Chair: Tina and Marcus, have you any views on that?
Tina Fahm: Yes. I have a point just to follow on from the optimism bias. In our sample of 24, we found that 22 had been revised down from the original target and only one had been revised upwards. That really causes some concern around the understanding about value, what the value that has been captured here is and how it is being managed.
We know that log frames are not static instruments and they are meant to be adjusted, so we welcome the new impetus around the amended annual review framework. It is important that the understanding around value and the value for money consideration is kept under close scrutiny. In our evidence, the value for money assessment as part of the annual review was, essentially, a narrative assessment. It did not robustly go back to the considerations and the assumptions made in the business case. I am delighted to hear now from DFID that that will be encouraged in future. Unless that understanding of the value that has been gained is kept under close wraps, value for money becomes meaningless.
Q14 Chair: I think that 22 out of 24 sounds a lot to be scaling down. This is a relatively random sample of projects. Is that typical?
Joy Hutcheon: It is not surprising that most log frames have changed, because when plans contact with reality, in the sort of countries we work in, you cannot know everything about a programme when you are designing it. The report does not go into the scale of the change, whether these are tweaks or radical changes. I talked earlier about the value we set on the objectivity and the robustness of the annual review. We understand that, if people are worried about this area, this is something that potentially undermines that, so it is very important for us. It is something that we want to keep a close eye on and look at further.
One of the reasons we have introduced the tracking is so that we have better visibility of the cumulative changes to log frames. They will be tracked in every annual review. You will be able to open something that shows you every change to the log frame. That will be published with the annual review. We have asked the quality assurance unit, as they do their spot-check of annual reviews, to really focus in on this issue. That will just make sure that we have the assurance we need and can give you the assurance you need that this is not undermining the annual review process.
Q15 Chair: As you say, you are not going to be able see if it is a tweak or a major change and these sorts of things. Does that feel right to you, that sort of percentage, that 22 out of 24 are going down, when you scale it up to the rest of the projects that DFID runs?
Joy Hutcheon: I would need to see how much they were going down by. I am not surprised that those log frames have changed. Often what you will see in a log frame is a shift between outputs, so you find that you can do more of something and you might therefore want to do less of something else, within the envelope of the programme.
Q16 Chair: To what extent do you think optimism bias kicked in? What sort of scale was it?
Joy Hutcheon: We are still in the process of dealing with optimism bias. Teams are very ambitious and very enthusiastic. They want to make a big difference and they want things to happen, particularly on timeframes. I do not think it is always that they think they can deliver more, but they think that they can deliver it more quickly. We are really working hard on trying to make sure that teams try to aim off for realism and aim off for that overenthusiasm. Again, this is something that the quality assurance unit challenges on our major programmes when they look at them.
Stephen Twigg: It does suggest an overwhelming optimism bias if it is 22 out of 24 cases.
Chair: This is what worries me, yes.
Q17 Stephen Twigg: Is it possible for Tina or Marcus to clarify how far these were tweaks and how far they were much more significant changes?
Marcus Cox: We would have to get back to you with the details; I do not have it at my fingertips. We would certainly welcome this tracking system. This is a rich area for DFID to enquire into in more depth. As Vel mentioned, the delays in programme start up are probably a significant driver in this, alongside the optimism bias question. If a delay in the completion of the procurement means you have lost a year of your programme life, you are necessarily downgrading your targets.
Joy Hutcheon: It is probably a good idea for us to have a look at ICAI’s data on this and see what level of issue it is.
Q18 Mr Lewis: I do not think we would want you to knock this sense of optimism out of DFID. Some of us know how that feels. The thing I would say is that surely it is not just about subjectivity; it is about a framework. It is about it being clear that there is some quality control of the framework that is applied to reach those conclusions. It is not entirely about the whim of individuals, or their overzealousness or enthusiasm. It is about a credible, robust framework that tests or helps people to make those decisions at the beginning of a programme. Does that exist?
Joy Hutcheon: Yes. When you are designing a programme and specifying outputs, we require teams to look at all the evidence and benchmark other programmes in that area and of the same kind, and look at the likely cost of delivering outputs, timeframes and do-ability. There will also be a lot of factors that are unique to the particular environment they are designing the programme in. It may be that those are ones that are harder to benchmark. For example, how fast will you be able to move with a Government champion on a programme that is seeking to reform accountability institutions? It is possible that it is those sorts of factors that teams are being overoptimistic on.
Q19 Stephen Twigg: I will move on to an issue that ICAI reported on: the catalytic effects of UK aid. Their review states that, as things currently stand, the results system is not oriented towards measuring or reporting on the contribution of UK aid to catalysing wider development, including country partners financing and delivering interventions themselves. Can I ask DFID to comment on that? In particular, how could that catalytic impact of UK aid be better reflected in the value for money framework?
Joy Hutcheon: We have a number of programmes that are setting out specifically to deliver transformational impacts. We talked a bit about those. Where we find this more difficult is working on wider policy issues. This is akin to Ministers making policy choices, if you like. Should we invest in a girl summit to address child marriage and FGM? Should we put our effort into humanitarian reform? Should we put it into UN reform? Should we put it into other areas?
The cost here is largely staff time and opportunity costs. If we put all our effort into a girl summit, is there something else that we could have been doing? Obviously those are judgments and trade-offs that Ministers are closely involved in. We will try to be clear about what impact we think can be achieved, what the theory of change is and whether there is someone else—whether we need to do this or whether, if we do not do it, someone else will.
Once the judgment has been made about what we are going to do, we try to be as explicit as possible about what we are trying to achieve. I do not know that any donor has really cracked being able to quantify and track impact through that kind of work, but that is something that we think about and will continue to challenge ourselves on.
Q20 Stephen Twigg: Tina or Marcus, is that responding to the point you were making?
Tina Fahm: Yes, it is. It is important, because clearly for DFID, in terms of value for money, it is very much immediate, quantifiable impact. What we are looking for is the transformative, and this refers to what we are saying in our first recommendation. That is very much about involving the countries in which we are working and the country offices much more. Again, it is being clear about what the level of ambition is. In other contexts DFID has been very clear about its level of ambition in economic development, in terms of climate change resilience. It is really tying this across to value for money so that it stacks up.
Marcus Cox: The missing piece here is the reporting of results at the country portfolio level. As Joy said earlier, the pattern in the last few years has been that country offices report on their contributions to global results targets. That was the priority at the time. What seems to have fallen away, for one reason or another, is this richer reporting against the more cross-cutting objectives of the country portfolios. When you read country operational plans, they talk about the importance of things such as promoting stability or promoting economic growth and economic transformation. There is not really a process whereby they go back and report on how the portfolio collectively is delivering against those things.
When you look at some of these ambitious goals, for example in DFID’s economic development strategy, which was released last year, that talks about trying to industrialise and create economic transformation in developing countries. These are things where you cannot really measure them by the aggregate results of individual programmes. It is important to measure how many microcredits DFID provides or how many jobs are created in CDC programming, but that, in itself, it not capturing that transformative goal. You need another level of reporting over and above that.
At the moment that strikes us as one of the key gaps in DFID’s results system. The next planning round in the spending review will be an opportunity for DFID to lift its level of ambition in this area.
Q21 Stephen Twigg: It makes sense to ask for DFID’s response to what Marcus just said, but to slightly word it in my own way. The review highlights examples where centrally managed programmes and in-country programmes might contradict each other and then undermine value for money. First, do you accept that is an issue? Do you accept what Marcus has just said is an issue? If you do, how do you propose to avoid this in future?
Joy Hutcheon: We agree that the portfolio management level is absolutely something we should pay attention to in planning for the next spending review, for all the reasons that Marcus gives. That will also include making more progress on centrally managed programmes. Four or five years ago, we had a position where we had not done enough to think about how our centrally managed programmes were integrating with our country programmes. We were thinking very hard about scaling up to 0.7% and delivering the aggregate results we wanted to deliver.
Since then we have taken a number of steps to ensure much better coherence between country and centrally managed programmes, starting by providing much better management information. Anybody in a country office can go on to our aid management platform, have a look and see all the programmes that are operating in their country. We have made big advances on geocoding, so you can see where they are operating in your country at a sub-regional level. The delivery chain tracking has also helped. You can actually go into that programme and see what the delivery chain is and see who is doing what in what sub-region in your country.
Most country programmes have now prioritised the programmes that they need to engage with. They do not need to engage with all of them. It may be that the International Citizen Service, or the support we give to the UN research, the CGIAR, is not something that is fundamental for that programme to engage with. Typically, offices have prioritised. They have appointed a contact person for each centrally managed programme. We have a coherent smart guide that sets out the expectations now of centrally managed programmes and country programmes in working together.
Q22 Stephen Twigg: Is there any response on that? You are nodding, Marcus.
Marcus Cox: We recognise some of those positive developments and mentioned those in the report. Certainly, there is now greater visibility on the net programming between centrally managed and in-country programmes in any given country. Indeed, we have seen the guidance for the design of new centrally managed programmes to pay more attention to coherence with the country programming.
We have a very long tail of centrally managed programmes that were developed in the past. It will probably take some years to clear out that backlog. In the meantime, it is something that has come up in multiple ICAI reports over the years, and indeed, I think, in some of the IDC reports as well.
Stephen Twigg: Yes, it definitely has.
Marcus Cox: There are value for money risks around a lack of coherence or even potentially acting at cross purposes between those programmes and the in-country programmes.
Q23 Mr Lewis: I have one observation very briefly on Stephen’s question. I suppose this is for ICAI, in the sense that I would guess that when there are political priorities and Ministers’ priorities, there will be an over optimism from civil servants, because Ministers demand and expect an over-optimism from officials on their political priorities. You might want to look at that dynamic in the future. We were Ministers, and that often happens, if we are frank. We are sat here, interrogating officials, but where politicians publish strategies they often want incredibly ambitious targets, whether they are realistic or not. Let us say they are very subjective in the judgment of what is realistic and achievable. It is something that ICAI might want to look at, in terms of political priorities.
I have a question for Joy and Vel. ICAI suggested that, when looking at value for money, there should be a greater focus, in terms of the definition of value, on the hardest-to-reach groups. For example, I welcome the fact that successive Secretaries of State have prioritised disabled people. When I was at DFID, if I mentioned it they said, “We have enough to do without worrying about that.” They did not quite say it in those terms, but more or less, so I welcome the fact that has been given political priority. Are you considering that, looking at all the programmes and saying, “We will judge value”—disproportionately, I guess, as that is what ICAI are suggesting—“by success with the most difficult-to-reach groups”?
Vel Gnanendran: Again, we absolutely recognise this. It is why we do not just use the three Es: economy, efficiency and effectiveness. We have the four, with equity on the end. We completely recognise that the value for money equation is not just about the greatest number of people at the lowest cost. There are very strong reasons why you might want to do something that costs a bit more because it is going to reach harder-to-reach groups.
We have a new guide on equity, the fourth E. That very clearly tries to articulate how teams can factor in that reaching marginalised groups into the value for money calculation. That now exists and we are seeing it in practice. In my last role in Tanzania we were looking at a family planning programme. It was much more expensive to reach women in rural areas than women in urban areas, but we still decided to try to reach those women in rural areas, even though the unit cost was higher. That is happening.
We need to keep driving that. As we really take on the “Leaving no one behind” and disability agenda, we need to keep driving that. I feel quite comfortable that our teams have all the support and the right incentives in place to make sure they can consider marginalised groups in the value for money equation.
Q24 Mr Lewis: Delivering against targets sometimes encourages people to go for easy, quicker wins, and easier groups of people. This is not a new debate about targets and performance indicators; it has been going on in Government Departments for decades. Is that not a risk as well? If people are focused on delivering the promise that they are going to deliver X, Y and Z, if you are going for more challenging groups of people, that makes it more difficult.
Vel Gnanendran: That is a risk. Now that we do not have fixed targets for, say, the number of children in school, it is less of a risk for us, so we can ask our teams to focus on reaching more marginalised groups. Again, in Tanzania we shifted the education programme to focus on children with special needs. That was partly because that was what the Government in Tanzania wanted to do, but also because the Government here wanted to prioritise that. The move away from fixed targets has helped us in this space.
Q25 Chair: We will start looking at annual reviews. ICAI argues that, during the life of DFID projects, the continuing validity of VfM propositions in the business case is not routinely checked. What do you say in response to that?
Joy Hutcheon: This is one of the things that we have strengthened in response to the report and through our review of annual reviews. There is this very explicit question now to SROs to provide an assessment of the value for money of the programme, with reference to the business case, and to then say whether and why the programme should now continue, on value for money grounds. We have been asking teams to do that. We have now stepped up and are asking them to focus on it a bit harder.
Q26 Chair: Does that reassure you, Tina or Marcus?
Tina Fahm: It does. The fact that the annual review framework has been reviewed and strengthened is very much welcomed. We will need to see it happening. Clearly, that value for money consideration is key to the annual review process, and it ought to be more than a narrative assessment. It really needs to go back to those initial assumptions. Assumptions may change, situations may change, but it needs to be a lot more robust than what we have found during the review.
Q27 Chair: How have you made that more robust? You say you have gone back and asked the teams to do these sorts of things. How have you taken that on board?
Joy Hutcheon: The teams are asked that question. That will be spot-checked by the quality assurance unit, and teams do not want to get a bad spot-check on an annual review from the quality assurance unit. For the reasons that Vel gave earlier, we do not think we should be specifying points at which an economic appraisal should be rerun. You may end up doing a lot of work rerunning an economic appraisal on a programme where it is not required, and you might think, “We do not need to do it for two years,” on a programme where it is required.
One of the things that I am attracted to, reading the report and thinking about it, is when you are writing the business case and preparing the economic appraisal, the person doing that might want to track assumptions that they have spotted that they think might be volatile during the programme or might be subject to change. They might want to flag to future SROs to keep an eye on this assumption because it is so critical to the economic appraisal and if it starts to move they would really want to rerun the whole thing. We may be able to ask the design team to provide some pointers to future SROs about the vulnerabilities in an economic appraisal that would help them in making that judgment.
Q28 Chair: In eight out of 24 projects that were reviewed, some VfM indicators were not being monitored as planned. How do you justify that?
Joy Hutcheon: I was concerned when I read that. It is not something that has come out of the quality assurance unit spot-check. The ICAI report does not say whether this is related to other changes in the programme or the environment. Again, this is something that we are going to ask the quality assurance unit to have a particular focus on as it does annual review spot-checks this year. They will be asked to just look at whether the right value for money indicators for the programme now are being assessed in the programmes that it is looking at.
Q29 Chair: ICAI said also that there was a pressure on staff to proceed and assessments appear to have been written in order to make a case for proceeding, in terms of annual reviews. It starts to reflect what we were talking about with changing targets as well before. What do you say in response to that?
Joy Hutcheon: Again, it is not something that has come out to us through our 10% check of annual reviews. If that is a perception that ICAI has through the 24 programme sample it has done, that is a matter of concern and we will be asking the quality assurance unit to look at it.
Q30 Chair: Do you have any views on what you have heard?
Tina Fahm: It is to be welcomed. It is a positive upward trajectory, although there is scope for improvement. DFID has clearly got the baseline arrangements in place, but now it really needs to be much more effective. We have heard today that arrangements are in place for that.
Marcus Cox: It is probably fair to say that our sample of 24 programmes extended back to programmes that were designed seven or eight years ago, when some of these value for money technologies were still in their infancy. It is important to say, as Tina says, that there is a trajectory of improvement there.
Q31 Stephen Twigg: Can I ask about the issue of programme sustainability? ICAI, in its review, referring to the water and sanitation programming, says it found that, unlike some other donors, DFID did not monitor whether results were sustained beyond the end of the programme. We know that DFID has since encouraged its implementers to increase their focus on sustainability, but did not take up ICAI’s recommendation of extending monitoring beyond the programme cycle. Can I ask why DFID has not been willing to do that?
Joy Hutcheon: We are very concerned to make progress on sustainability. Again, it is fundamental to impact. We welcome the challenge that we have had from ICAI and this Committee on sustainability over the last few years. As I said earlier, there are some programmes where the evidence as to the impact of the thing you are doing is very strong. If you educate a girl, you know that she will be healthier, her family will be healthier and she is more likely to educate her children. She will be better off. You know that if you vaccinate a child you are going to protect that child. You know that if you get somebody sleeping under a bed net they will not get malaria. You do not need to go back and check what the outcome was at the end of that programme.
There are other programmes where the relationship between output and outcome is not as well evidenced. There are cases where we do go back and do that checking. We are about to launch a review of the outcomes of that water and sanitation work between 2011 and 2015, following the review that ICAI did. We have a programme on preventing violence against women and girls, where the evidence is again quite weak as to what works. That is running an intervention for three years and then checking the outcomes for the subsequent two years.
In some cases, that link is being tested elsewhere. For example, on social protection programming, there is a body of random control trials that are evidencing the impact that social protection has and the differential impact that differently targeted social protection has. That means that actually it would be duplicatory for us to go back at the end of all our social protection programmes and run a very expensive and labour-intensive RCT. We do this sometimes, but we do not do it everywhere.
Q32 Stephen Twigg: You gave an example of girls’ education and the wider effects as where things are well evidenced. Even in that example, would there not be a case to go back once a programme had finished, for example to see, despite our programme finishing, whether support for girls’ education in that country has been sustained, or what the impact has been on the life chances of the girls or young women that have been part of that programme. Even in the well evidenced cases, is there not an argument to be made for going back and monitoring beyond the programme?
Vel Gnanendran: Just to start on that, whenever we close a programme we do a completion review. At that point, if the programme has been going for four years, we do a completion review so we can start to see what some of those impacts will be. I guess this is a question about evaluation further along, which we do on a very ad hoc basis. If we are trying something new, usually there will be an evaluation. If we are trying something in a new environment we might do an evaluation. As Joy was saying, where there is a fairly safe intervention in a fairly safe place, a fairly safe country, we probably would not prioritise that for the kind of work you are talking about. We would want to prioritise the evaluation somewhere else where there is not as much learning out there that we could draw on.
I just want to make a point on financial sustainability, if I may. This is an important question, the degree to which programmes are financially sustainable after we exit. One of the reasons we take a slightly different approach to some other donors is we tend to stay in for the long term. We might have three or four generations of programme supporting a country’s social protection programme. There, each successive business case will see how much the Government contribution has increased.
As Joy was saying, across lots of social protection programmes in Africa, we are seeing that we can invest less over time because the Government are investing more, because we are in there for the long term. We are seeing that in Kenya, Rwanda and Ethiopia. In Kenya, in fact, I think we have come out completely from the social protection programme because the Government are fully financing it. That financial sustainability question is partly we can just stay in for a bit longer, and each time we are staying in we are judging whether the Government are meeting their contribution.
Tina Fahm: This is an important point that speaks to value for money; not simply monitoring and evaluation but, having made the initial investment, at the end of the programme to what extent is value still being created and delivered. I would strongly endorse the finding that we make in our report, notwithstanding the different scenarios that DFID has presented. The focus here is value for money and what we want to see is how that accrues or delivers over time. We need some mechanism that is able to capture that.
Q33 Mr Lewis: This question is again for Joy and Vel. A key element of value for money in development programming is alignment with the priorities of the Government of the country. Obviously, DFID no longer does general budget support, and that therefore inevitably changes that relationship. Do you think the ending of general budget support has undermined that alignment, in terms of value for money?
Joy Hutcheon: One thing it has made us do is think very carefully and explicitly about it. Where we can see an alignment with Government objectives around something particular, it is still open to us to provide financial aid and work through Government systems. We are thinking very carefully about where that is the best way to deliver a specific objective. Where you do not have alignment on an objective with Government, certainly the evidence indicates that you are much less likely to deliver that, particularly transformative outcomes, by working through Government systems. Also, in many cases—this is really one of the reasons for moving away from budget support—we are confident that the risk to value for money of putting money through Government systems was reducing.
Q34 Mr Lewis: In terms of that alignment though, you said where there is not an alignment. I thought that the whole point of a country-led view of the world is that there will always be alignment.
Joy Hutcheon: I am trying to think of the best way to answer this. Where a country is genuinely implementing poverty reduction objectives, there is likely to be an alignment with things that we are working on in that country. There are some countries where either the Government objective is not to deliver those poverty reduction objectives, or where their systems are sufficiently weak that it is not the best way for us to pursue them.
There is an interesting example in Uganda, where we are no longer putting money through Government systems. We wanted to work with the Government on social protection, so there was alignment, or we hope there is alignment, with the objective. We have been co-funding a social protection scheme, a senior grant, and the plan is for the Government to take that over when our funding is withdrawn. At that point it will be clear to what degree that alignment has been there.
Vel Gnanendran: I was just going to iterate that point about the distinction between alignment with Government objectives and use of the Government system. Those are two different judgments that we make. We might be aligned but choose not to use the Government system because we do not trust that Government system has enough fiduciary safeguards. We would never operate through Government systems where we did not feel aligned, but we can be aligned and not work through Government systems.
Q35 Mr Lewis: How does ICAI feel about DFID’s response, in terms of better applying development effectiveness principles, including the point they have just made about aligning with individual countries’ priorities?
Marcus Cox: It is certainly a finding from our review that development effectiveness principles are not very visible in the way DFID approaches value for money. Having been through this shift in policy away from general budget support, it is no longer front and centre in the way programmes are managed. We had a concern from national stakeholders that we spoke to over the course of this review that DFID was increasingly going off to do its own thing, focusing its attention on the delivery of its own programmes and less visible in terms of the support for Government.
That is a perception. That is certainly not universally the case. Certainly, development effectiveness is still part of the way DFID staff understand good development practice. In some ways, it is part of the DNA of the organisation, but it is not very visible within the debates on value for money. Aside from the question of whether you are choosing to put money through country systems, there are questions around whether you are putting the right level of effort into working and engaging with national stakeholders, even if that means higher management costs or higher unit costs in your delivery, in order to make sure that you are building sustainable national capacity.
What we have challenged DFID to do is to think harder about how to bring those questions in. The response, for the time being, is a bit non-committal on this point. We would be hoping to come back in 12 months and see some more specific ideas about how to address these things. One of the areas that we would be keen for them to explore is bringing in the quality of delivery alongside the quantity of delivery. That is paying attention in log frames, targets and value for money assessments not just on how much is delivered but also how it is being delivered and the extent of engagement with national actors and capturing that transfer of capacity to them.
Joy Hutcheon: We have accepted this recommendation. The Smart rules require us to always think about development effectiveness, but we need to make sure that we are thinking explicitly about this. We are about to participate in a pilot with the Global Partnership for Effective Development Co-operation. We are talking to other donors about how they are integrating development effectiveness into their work. Again, this will be something that we want to think about as we go into the spending review and planning the next round of the portfolio.
Q36 Chair: Stephen mentioned earlier the dichotomy with performance‑based conditionalities with multilaterals. Is that the way you would pose those performance-based conditionalities? Does having that approach have the potential of damaging your relationship with multilaterals?
Vel Gnanendran: It is always challenging when you impose these sorts of conditions. The response is different. For example, in our first performance-based tranche to the WHO, we put in there a condition to have an organisation-wide VfM strategy. Actually, they responded to that very enthusiastically. They have taken that on board and we are able to use that at country level now. The story is very different.
I think it would be fair to say that the multilateral system would prefer to have completely unconditional funding, so we obviously expected some level of pushback. For the UN performance-based funding, up to 30% is performance based, so it is not the whole lot—70%-plus is still core funding. That is just part of the relationship. It is a strong relationship that we have with all the multilaterals. It is a grown-up relationship and we can have that discussion with them, but we expected a level of pushback and we get some of that.
Joy Hutcheon: We have challenged the multilateral system a lot over the past five to eight years, including with the multilateral development review, removing funding from lower-performing multilaterals. Any of that could have damaged the relationship. What is key is the multilaterals seeing and understanding that we are trying to improve development effectiveness and impact and reduce poverty and that is where we are coming from. That is what enables us to maintain the relationship.
Q37 Stephen Twigg: Could I ask a follow-up on that before my question? I do not think anyone doubts the motivation. It is about unintended consequences. The example of saying to WHO, “Have an organisation-wide value for money review” is a positive example. What I sometimes pick up from multilaterals is the sense that, at a board level, one set of outcomes will be agreed, and then DFID comes in and says, “We want you to do these other things,” which were not the things that were agreed at the board level. Do you think there is some validity in that concern, particularly for UN agencies?
Joy Hutcheon: We try to avoid that outcome by talking to other donors and talking to like-minded donors about what the donor community agrees should be the reform objectives for UN agencies. I guess we will not always agree with everything that agency then decides at its senior level and we may keep putting pressure on them. We try not to just do our own thing alongside lots of other people doing their own thing.
Q38 Stephen Twigg: Am I right that we are the only donor that has this kind of conditional funding? No one else goes to UN agencies and says, “We will provide this funding only if you meet X, Y and Z conditions”?
Joy Hutcheon: I think there are a number of other donors who are looking with interest at what we are doing and thinking about whether they want to do it. I cannot tell you today whether any of them have gone down that route or not.
Vel Gnanendran: I am afraid I do not know the answer. This is complete speculation, but I suspect there may be a free-rider problem as well, in that they do not necessarily have to do it if we are doing it.
Q39 Stephen Twigg: My final question is to ICAI. In the review, you raised a number of interesting issues. I will mention some of them. They are ones we have covered today. You raised them but then did not make any specific recommendations. For example, there is the issue that I mentioned earlier of the catalytic effect of UK aid, the benefits of weighting different classes of beneficiary and the risk of centrally managed programmes and in-country programmes contradicting each other, which we have just talked about. Can you explain why you did not make specific recommendations on those issues?
Tina Fahm: Yes, of course. The report and the five recommendations we make address the most significant findings and weaknesses that arose from our review. The other points, in terms of weighting different classes for beneficiaries, aligns very much with the “Leaving no one behind” commitment and the fourth E, equity. We are pleased at the progress DFID has made there, if you will remember that it came up in the marginalised girls review. We noted the progress there. I will hand over to Marcus to perhaps say a bit more about weighting different classes of beneficiaries.
Marcus Cox: Specifically on the weighting of beneficiaries, as Vel said earlier, it is listed as a possibility in DFID’s new guidance on equity and value for money. While we think it is an interesting idea, we are not necessarily of the view that it is the universal solution. It is one of the tools in the kit by which you can start to wrestle with the application of equity and the “Leaving no one behind” commitment in value for money, but not necessarily something you would want to see as mandatory. The fact that it is there in the guidance is already a positive thing.
On the centrally managed programmes, our finding in the review was that there was broadly a positive direction of travel here. We welcomed the new guidance on coherence between centrally managed programmes and in-country programmes. We also welcomed the development of a new generation of programmes that we think approach this coherence issue in a much more structured way. While there is this long tail of things to go through, there are measures in place to try to make sure that future centrally managed programmes are more appropriate.
On your first point on the catalytic effect one, our first recommendation is very much intended to address that. Our first recommendation is about articulating drivers of value for money at country portfolio level and reporting against those. That is what we would see as the solution in that case.
Chair: Thank you all. Thank you to the ICAI for a very well written and interesting report; it is very useful. Thank you to the four of you for engaging so well on what is such an important process. We know that value for money is important across every Government, and across any Department. Especially when we continue to make the case for international development, it is really important that we keep on our toes for value for money. I should put on the record the work of my predecessor, Fiona Bruce, who pushed for this report to be written for value for money. Thank you so much for your time this morning.