Sub-Committee on the work of the Independent Commission for Aid Impact

Oral evidence: ICAI’s Review on Water, Sanitation and Hygiene, HC 102
Wednesday 29 June 2016

Ordered by the House of Commons to be published on 29 June 2016.

Watch the meeting: Wednesday 29 June 2016

Members present: Fiona Bruce (Chair); Mrs Helen Grant; Jeremy Lefroy; Stephen Twigg

Questions 1-47

Witnesses: Dr Alison Evans, Chief Commissioner ICAI; Richard Gledhill, Lead Commissioner, ICAI, and Nigel Thornton, Team Leader (Agulhas Applied Knowledge), ICAI, gave evidence 

Q1   Chair: Good morning and welcome to our first meeting with you, as a Sub‑Committee of the International Development Select Committee. I would be very grateful if you could, first of all, introduce yourselves. Thank you, Dr Evans.

Dr Evans: I am Alison Evans. I am the Chief Commissioner of ICAI.

Richard Gledhill: I am Richard Gledhill. I was the commissioner who led this review.

Nigel Thornton: I am Nigel Thornton. I was the team leader on this exercise.

Q2   Chair: Thank you for coming to be with us today. Could I ask you, firstly and fairly briefly, why you chose an impact review, as opposed to a learning review or a performance review, and what your objectives were in selecting it?

Richard Gledhill: Yes, certainly. The WASH review was ICAI’s first pure impact review. When the new team of commissioners came on board we decided that, as well as looking at performance reviews and learning reviews, we wanted to have a formal process of impact reviews, where we look at the results claimed for more established aid programmes to assess their credibility and significance for the intended beneficiaries.

We chose WASH, because there is a growing level of expenditure on WASH programming, both in money terms and as a percentage of UK ODA. In the five years to 2014, the UK spent over £700 million on WASH and, by that year, it accounted for 1.6% of UK ODA. We also chose WASH because it is such an important part of the international development agenda. Access to water and sanitation and improved hygiene are obviously vitally important in their own right, but they can also contribute significantly to improved education, better nutrition and greater gender equality. From that perspective, we thought it was a good place to start with our first impact review.

Q3   Chair: Thank you very much. You say that you find DFID’s claim of reaching 62.9 million people with access to WASH as credible. The main purpose of an impact review is to examine results claims and their significance for intended beneficiaries, and we know that the DFID results framework refers to beneficiaries as the number of people with sustainable access to clean drinking water and sustainable access to improved sanitation facilities. How did you come to a conclusion on sustainable impact when your report, in a number of places, makes comments such as, “From the evidence available we cannot reach conclusions as to where and to what extent impacts are occurring”, “Impact data is not routinely available at the programme level”, “Many are not set up to generate the evidence that would allow this to be measured” and “We cannot reach conclusions as to where and to what extent impacts are occurring due to lack of data”? How did you conclude that the results claim was credible, if you did not have the data to measure impact?

Richard Gledhill: When we decided to look at the WASH programme, we looked at both the target and the claims made by DFID. The target was set as a reach target and the primary focus of our work was therefore to assess the credibility of that reach target. In doing so, we looked at the evidence of how reach targets convert into impacts, and there is good evidence of that in the literature in many areas. We then probed whether that evidence was borne out by the experience on DFID programmes and we found many instances where there was good evidence of impacts, and we have highlighted a number of those in our report.

              We found that DFID was not systematically measuring, recording or aggregating impacts, nor was it systematically looking at the sustainability of those impacts. In our overall scoring, we took account of the weaker performance, particularly on sustainability, and we made very strong recommendations in the report about sustainability requiring urgent remedial action, but we also included important recommendations in relation to impact.

Q4   Chair: On impact you actually gave a green/amber score. This is my question: how could you give what is a more positive score than the one on sustainability, when you did not have the data to reach a conclusion on it?

Richard Gledhill: The results lead to direct impacts in relation to WASH access. Where we felt there was less consistent evidence, it was in relation to the wider impacts around education and health and gender equality. We did not find evidence that this was not happening and we found many instances where it was happening. Looking at the overall result, we felt that warranted a green/amber, although clearly there is more to be done in relation to impact measurement and recording.

Q5   Chair: Your conclusion was that there was a lack of consistent monitoring on impact but that, where there was monitoring taking place, you felt that was acceptable.

Richard Gledhill: In many programmes, there has been monitoring and evaluation of impacts, but not consistently across the portfolio.

Q6   Chair: My final question is about whether you think that is acceptable that there is no consistent monitoring for impact on such an expensive scheme, £700 million over four years between 2010 and 2014?

Richard Gledhill: We believe that DFID should be doing more on impact measurement and recording.

Q7   Stephen Twigg: Can I pursue the data point a bit further? You said in the report that this was “based on population‑based assumptions about WASH access that are widely used by agencies in the sector”, as opposed to actual usage, whereas DFID’s management response refutes your report and says, “The number of people reached with sanitation and hygiene are not based solely on assumed numbers of users of service as stated in the report, but based on counts of people with access to toilets”. Can I ask two questions? The first is for you to comment on that disagreement between your report and DFID’s response. The second is more of a process point: considering that all ICAI reviews go through a fact‑checking with DFID, it is unclear to us how this discrepancy has come about. Could you explain?

Richard Gledhill: We were surprised by that response. We believe that our report is accurate and certainly this issue was not raised in the fact‑checking. Perhaps that is a point that might be picked up with DFID later. To some extent, the issue might be a slight issue of semantics, but perhaps I would ask Nigel to comment on the detail of the processes we went through in that area.

Nigel Thornton: Thank you. I think it was surprising, the management response. Sorry, it was not raised in the management response, but came afterwards. There is a series of issues. One is, when one goes into the field and one talks and sees how the numbers are aggregated, let us take the example of Mozambique. In Mozambique, the numbers of people who are served by a borehole is 300, according to DFID. There was a survey that was done by Oxford Policy Management, which is doing a piece of work on value for money, which found that it was between 120 and 160 people served by that. When we went into the field, we found that it was actually up to 2,000 people served by water point in particular locations.

Richard Gledhill: That was at a point in time.

Nigel Thornton: At a point in time. One can see that there is actually a huge variation in numbers that one could choose. The main assumptions that are being primarily used for access to water and also for sanitation are population‑based ones, which are saying there are a number of boreholes in this area; there is a number of population in this area. We divide one by the other and find out that that is the number of people being served, or we make an assumption based on the local planning standards, which might be 300 people per borehole in Mozambique or 200 in another country.

The essential point is that it is not being surveyed. There is no survey data. When one provides latrines, sometimes there is survey data being used, but not in all cases. Actually, if you look at the way the report is written, it makes very clear that it relies in the main on estimates. We do not say this exclusively in the report. Again, when DFID came back to us and said this in the management response, we were surprised because, one, we do not claim that in the report and, secondly, it was not raised during the fact‑check.

Q8   Stephen Twigg: Obviously we will take an opportunity in the second panel to ask DFID about that but, just to be clear, you said right at the end that it was not raised in the fact‑checking process with you by DFID.

Nigel Thornton: No.

Richard Gledhill: We are comfortable with the use of estimates in that way, and we believe overall that this has led to a conservative reporting of results. Where it does present challenges is when you look forward. Applying the new global goals in relation to leaving no one behind, the use of estimates makes it harder to look at individually more disadvantaged groups, such as disabled, elderly, women and girls.

Q9   Stephen Twigg: Another point but related is that you do not, as I understand it, carry out your own independent assessments of impact, which you state would have involved monitoring over an extended period. Does that mean that you are essentially very reliant on the data from DFID? You have just cited evidence from others that you have relied on as well. How far are you reliant in practice, because of your methodology, on data from DFID?

Dr Evans: This is a really important point. With all the will in the world, we would very much like to do very rigorous, thorough impact assessments on pretty much all of these results claims. Frankly, that is beyond our capacity and resourcing. They are very technically demanding and do require commitment over quite an extended period of time. Our entry point largely on this is about the extent to which DFID and DFID’s partners are investing appropriately in independent impact assessment or quality impact assessment that we can then verify and assess. In the context of this review, there are a number that we are able to look at and assess the quality of, but essentially a portfolio this large makes it almost beyond our capability to do rigorous impact assessment in every programming area in which DFID is engaged.

We very much take on the issue of whether impact assessments are present or not. How does DFID actually know about its results footprint in this particular programming area? What we are saying in this review is that, actually, there is a lot of impact, but it seems that DFID does not actually have a good way of capturing most of it. We are pretty confident; in fact, the team was confident that they had caught sight of a considerable amount of this impact in the two countries that they visited, but DFID is not systematically reporting that. There is a sense of, “Why is that?” and the report takes that on head on, and says that this is an area for some considerable improvement.

Chair: It is interesting that the National Audit Office, many years ago, said it is unlikely that sustainability would be achieved if it is not monitored.

Q10   Jeremy Lefroy: Good morning. You state on page 25 of the report, “However, the absence of value for money measures across the portfolio makes it impossible for either DFID or ourselves to draw conclusions about the overall value that DFID is achieving in its WASH portfolio as a whole.”  Why do you say that?

Richard Gledhill: As part of our work, we looked very carefully at DFID’s approach to value for money, both at a programme level and at the portfolio level, looking at the programme design, through the procurement, through management and reporting. As part of our work, we were able to look at the documentation in relation to the execution of those processes at the programme level and to engage with DFID staff and indeed with the programme implementers.

That was primarily done from a desk basis. We covered a sample of the portfolio and were able to drill down a little more deeply onto country visits. What we had hoped to find was also some portfolio‑level approach to ensuring value for money through use of benchmarks and metrics, and analysis of costs and performance. We did not find that systematic data.

              We did see that DFID recognises that this is a challenge. Indeed, we had a lot of conversations with other people in the sector, NGOs and private sector operators. This is a challenge for everyone in the sector. DFID recognises this and is doing research on this, and we commend that. We also believe that they need to move quickly on this.

We saw at the programme level a number of very good instances. We highlight in the report the experience with UNICEF in Zimbabwe. That is an interesting programme to look at anyway, because it was not competitively tendered, so that immediately puts your antennae up. We saw that, initially, the programme was performing less well than expected. DFID put in place a performance improvement programme, which ratcheted up the management and effectiveness of the programme, drove up value for money and led to a step change in results. We saw a number of instances comparable to that, which suggest that there are good things happening at the programme level, but we want to see more at the portfolio level. It was that that meant we were unable to make an overall conclusion.

Q11   Jeremy Lefroy: Would you not think that this is pretty simple stuff, in the sense that we are talking here about things where the costs are fairly obvious? I noticed that there is a report here on page 10, “The drivers of cost difference were found to be complex and multifaceted, including economic, social, demographic and hydro-geographical factors”. That is a statement of the blindingly obvious in that, if you are drilling a well in a place with a lot of hard rock, it is going to be more expensive than somewhere else.

Surely there would be the opportunity for DFID to go and make a very quick assessment of what is available in an area. Maybe there is nothing available; in which case, the costs are likely to be quite high. Usually these kinds of things like shallow wells have already been done on a private basis, or latrines have been done on a private basis. It is very easy to get those costs and then say, “Surely we can do better than that”. The second aspect is the long‑term nature of it. Is the work that is going to be done of a quality that is going to last two years, five years or 10 years? These are all very basic things that do not need large reports, but that can be done at a very basic level. I speak from a little bit of experience in this.

Richard Gledhill: I agree that large reports simply do not solve anything. What solves things is good processes, good data and sharing experience among DFID staff and between the NGOs and programme operators.

Q12   Jeremy Lefroy: You are saying that that is not there.

Richard Gledhill: We are saying that more needs to be done in all those areas. You have talked in particular about the long‑term costs.

Q13   Jeremy Lefroy: If I may come back on that, DFID has been working in this for decades, so why is it not there now? We are not talking about something that is incredibly complex. We are talking about some things where it would just take people to get together on work on it and yet we have spent hundreds and millions of pounds, and we still do not have a framework. I am not trying to suggest that you are going to have similar costs in different countries. Of course there are all these different factors but, in each individual location, surely we are talking about some fairly basic procedures.

Richard Gledhill: That is probably a question for DFID, but I would say that they had recognised this issue before we came on the scene. Work was underway and there are plans in place to implement those findings. To pick up on one point you mentioned, you talked about the long‑term costs; we think that is an important deficiency in the data that is typically used by DFID. We think they should be looking at the lifetime costs of infrastructure.

Q14   Jeremy Lefroy: That just brings me to my second question, Chair. Did you visit any programmes that DFID had done, say five or 10 years ago, to see whether they were still in operation and, if so, why that was and, if not, why not?

Richard Gledhill: Box 9 on page 23 gives an example of a programme that was finished in 2012, where DFID had funded the construction of latrines, hand‑washing facilities, tippy taps, and also training on hand‑washing through health clubs, which we saw can be very effective. Our sense from what we saw on the ground was that efforts to change behaviour need to be sustained over time to deliver lasting results. Here we found that the latrines were still in use, but the water supply at the schools we looked at, in many cases, was no longer functioning properly. The health clubs had ceased to operate and there was no evidence of hand‑washing.

There was a very mixed picture on sustainability there. There is more sustainability in the hard assets and the infrastructure, but less sustainability in behaviour. That is the challenge that DFID, and indeed other donors in this sector, face. Nigel, do you want to add on that? You visited the scheme.

Nigel Thornton: Yes, that is right. We did that and, for instance, we also looked at a large infrastructure programme, which was the Bulawayo rehabilitation programme, which was looking at the rehabilitation of a big water supply programme. That had sustained. The pattern is as Richard said, and that comes through also in DFID’s own evaluations. It comes through in the evaluation of the work that Richard has just quoted, in box 9 on page 23 there. One tends to see some of the infrastructure sustained and not see a behaviour change sustain. If there are no basic things in place around institutional capacity, basic funding and basic technical capacity, the infrastructure degrades very quickly as well.

We were interested in that. One of the key recommendations in the report is that DFID does those sustainability checks afterwards in a much more systematic way than has been done to date to find out whether the investments are continuing over time and seeing whether value for money is achieved, over a longer period than just the project life.

Richard Gledhill: When we met with the NGOs involved in this sector, this was the big topic of conversation. Everyone is grappling with it. Everyone is struggling with it. WaterAid is very clear that you have to have the institutional strengthening, the governance and the financial strength at the sector level to be able to sustain long‑term impacts at the programme level. DFID is doing a lot of that, but it is not doing that systematically across all its programmes and we think that is an area that needs looking at.

Q15   Jeremy Lefroy: If I can just follow up very quickly with the final point, did you get the impression that many of the programmes were demand‑driven. When I say “demand‑driven”, they were driven by the people who were the beneficiaries, or were they programmes that were devised from on high, whether through DFID, the national government or the regional government? They then just said, “Right, we are going to do this for you. Here you are”. The experience tends to be that those that are demand‑driven and that have real buy‑in from local communities will probably be more sustainable.

Richard Gledhill: Nigel, do you want to comment on your experience in Mozambique?

Nigel Thornton: Yes, sure. What is interesting is that you see both. You see programmes that have been developed closely in partnership with government. It is good old aid effectiveness stuff. If you have a good relationship with your partners, the beneficiaries, whether that is local government or the local communities, then you are going to have a better sensitivity to what the needs are and you are going to design it more effectively. We saw some very good examples of that.

We also saw examples of centrally managed programmes that had been let and contracted globally, across many different countries. The classic one we saw in Mozambique, where communities were being provided with sanitation and hygiene, but not with water. When we sat under the trees and talked to the communities, they said, “Actually, we want water as well”. When we talked to the programme implementers, they said, “It does not make sense not to do water”. When we talked to local government, they said, “It does not make sense not to do water”, but the project itself just provided sanitation and hygiene. That was not fully demand‑led. To be fair to DFID in Mozambique, they have now put in some remedial steps to try to fill the gap.

One sees both and it is on a case‑by‑case basis. I think we raised in the report that we did have concerns about centrally managed programmes not necessarily being designed with sufficient input from local offices or local staff. That was an issue, and there could be a trend, so that might be a risk in the future.

Q16   Chair: Can I just stay on value for money for a moment and ask about scoring? If I can just quote from the report on a number of pages, at page 24, you say, “DFID does not apply a consistent approach for measuring value for money across its WASH portfolio, nor does it have credible benchmarks” and “is not currently using convincing methods for comparing … costs”. Page 25, “The absence of value for money measures across the portfolio makes it impossible for either DFID or ourselves to draw conclusions about the overall value that DFID is achieving in its WASH portfolio as a whole”. Again, the same page, “data [for value for money] was not available for many programmes or where available, there were large variations across indicators between countries”. There are just a couple of others again on the same page, 25, mentioning large variations across indicators. There was reference on page 28 to “DFID not yet using a convincing methodology or consistent data to enable it to compare the return on its investment across WASH programmes to help identify inefficient programmes”. The reason I have picked out those comments is because you then go on to refer to value for money under your green/amber rating, under impact. Why did you not refer to it under the amber/red rating of sustainability?

Dr Evans: Before Richard responds and Nigel responds, can I make a general methodological point about this value-for-money thing? I am referring back to Mr Lefroy’s question too. The disciplines that go with thinking through value for money and the structured thinking that goes with that and so forth are very present within this WASH portfolio in DFID. A lot of it is undocumented, however. A lot of it does follow the lines you have just indicated, which are that we kind of know how to do this. We have a sense of what the cost‑to‑benefit ratio is and we can go ahead. We have thought through how to minimise costs and how to maximise results.

What we are pointing to is a bigger challenge, which is systematically documenting the value-for-money calculus that is used across the portfolio. At the moment, there is no common standard for that, so we have data in some places and not in others. We have unit cost data that is calculated in some parts of the portfolio and not elsewhere.

That does not mean, as I started out, that the discipline of thinking how we achieve maximum value for money here through this programming is not taking place, but our challenge back to DFID is to invest very quickly in more systematic data to allow a much more consistent and rigorous assessment across the portfolio as a whole. It is not that it is absent; it is not that we do not think they are considering value for money. It is that, actually, as yet, they are not doing that consistently and systematically through the data that they are able to gather. That is the general point.

Richard Gledhill: The question about why value for money was not referred to under sustainability is a good question. When we looked at the framing of the questions but also the scoring, we were alert to the danger of duplicating a negative or a positive by reflecting it in more than one score and therefore giving undue balance to that area. Arguably, VFM is an important part of all aspects of the questions we looked at and we sought to focus the scoring of it more narrowly to avoid duplication.

Q17   Chair: It sounds like you had quite a fine judgment to make there between the scoring methods. Thank you. Can we turn now to sustainability and your amber/red scoring? Your review does contain serious concerns about the sustainability of DFID’s work in this area. Some of your comments are that DFID is not measuring and reporting sustained access. It has “no systematic evidence” to show sustainable results. “Attention given to … sustainability varies from programme to programme”. Programme length is often “too short to ensure sustainability”, and extremely concerning to some of us here on the Committee is that there has been little or no improvement since the National Audit Office’s 2003 review, which you quote saying the NAO then found “a lack of available evidence to assess the extent to which DFID’s projects are achieving a lasting beneficial impact. Half of the available assessments concluded that it was too early to judge the likelihood of sustainability and, of the remainder, two-thirds of reports raised doubts and risks as to whether a sustainable impact would be achieved”. Thirteen years on, we find that the situation has not improved substantially. Given the importance of sustainability to WASH, why did you then give an overall green/amber score?

Richard Gledhill: We were very clear that DFID needs to do better on sustainability and we gave that an amber/red score in the detailed scoring. We also gave what we think is a very strong recommendation in this area, but the points you have highlighted are very much or primarily about not having the evidence, not being able to demonstrate, rather than having evidence that sustainability is not there. We highlight some examples of where sustainability is not in place, but we also highlighted that this is a challenge across the sector and other donors and implementers are facing the same problems.

We did not wish to judge DFID against an unrealistic target, but we are not underestimating, in any way, the importance of this area. We are very clear that this is not a good result and that DFID needs to do more in this area. We are slightly disappointed by the management response on this. We would expect to have a clearer commitment from DFID, as to the actions they are going to take and the timescale with which they are going to do that.

Chair: We will come on to that in our final question. Before we do, Helen has a question. It is interesting that you refer to other donors facing the same challenges, but you also say that other donors are in fact making greater strides in this area. I think, Helen, you are going to ask about this.

Q18   Mrs Grant: Thank you very much, Chair. The review points out that USAID and the Dutch development agency check sustainability for up to 10 years after the end of WASH programming. How do their sustainability checks work and how onerous are these? Also, why do you think DFID has not learned from these leading examples and how should they really proceed to embed sustainability in its WASH programme?

Richard Gledhill: Nigel, do you want to go through the details of what USAID and the Dutch are doing?

Nigel Thornton: Sure. The Dutch include a clause in their contracts with implementing agencies. That clause, although we could not find an example of it being invoked, would withhold some payment or even ask for payment not to be completed until there is a demonstrated sustainable delivery over a longer period of time. Their sustainability checks are carried out both during the period of implementation and afterwards. For each individual project it varies when that is. It could be two years; it could be three years; it could be five years; it could be 10 years, but it is essentially after that period. It depends on the individual context.

The checks are conducted annually or half-yearly and they are usually done by a third party, which then reports. They take place in a rather similar way to something that DFID is working on in its Payment by Results programme, with randomly set checks for communities. Is the work achieving what it was planned to achieve? A third party goes in, selects where it is going to look and does the check. There is a set of variables: the actual specific project outputs, whether a number of water points are still working and they also look at the factors that support sustainability—the institutional arrangements and the financing—and whether those are in place.

USAID, again working with a UK firm—and there is a lot of thought leadership coming out of the UK in this, which is interesting—developed a sustainability index, which they piloted and have used to test each of the projects against a set of criteria. It is rather similar to the Dutch approach. It looks at whether the mechanisms are in place to ensure sustainability. They did that with Rotary International as well, in the Dominican Republic, Ghana and Philippines. There is quite a lot of thought leadership outwith DFID that DFID could draw on in this. The question about why DFID has not done that is a question for them. I am not sure that we can answer that.

Q19   Mrs Grant: Do you think they should?

Nigel Thornton: Yes, we do. There is a technical project management challenge, which is that, when one talks to staff in DFID they say, “We do not have a mechanism for funding this outwith the five‑year or three‑year period of a project. We just do not have that.”  One can always find mechanisms, we think, but at the moment the response is a kind of bureaucratic one that there is not a way of doing this. As I said, that might be something to test with them

Richard Gledhill: On the issue of sustainability, we need to look beyond just measurement and reporting. It goes back to programme design and programme duration, and not just at the local level, but also engendering institutional development and systematic change that will support local‑level sustainability. It is a bigger issue than just reporting.

Nigel Thornton: If I may add one final thing, DFID’s line on this, which they may say, is, “Actually, what we want to do is get our partner Governments to do this”. That is great and actually a lot of investment has gone into partner Governments to build their information systems, but DFID is not then mining that, going back when it has done that, finding out and doing those checks itself. It is not using the information that is available to them. We are not saying that DFID has to have third‑party monitoring hired in; we would just like to see that happening over a period of time.

Dr Evans: I was going to add very quickly that the challenge around sustainability is that sustainability has been on the agenda in this sector for a very long time and it has had a rather output focus. What we are seeing and what we have seen over the last number of years is for that now to expand, taking in sustainability around the outcomes and impacts associated with those outputs. That is a challenge. That is why we say that this is a challenge for everyone. In that sense, the ambition about being able to see impacts endure and accumulate over time is where the challenge lies, and that is where we feel that DFID is not as well positioned as it should be to be able to think about maximising impact over time, through being able to monitor it, measure it and know what good looks like. That is where they are not well positioned.

Q20   Mrs Grant: You have touched on my next question already, in relation to the response on sustainability, but are you satisfied with the management response that you have received generally from DFID?

Dr Evans: No is the short answer. A more expanded answer is to say that we are actually quite disappointed and we felt that we pitched a number of really quite strategic recommendations and felt we got quite a lot of narrative about everything that is already being done, as opposed to taking on board the substance of those recommendations. I will let Richard talk about the specifics.

Q21   Mrs Grant: Are there some specific areas where you really are extremely disappointed?

Richard Gledhill: Perhaps I can come back to that in a moment, but I will just explain how we have worked with DFID on this review. We have sought to be really open and transparent in our dealings with them, through the approach phase, but also reporting back to them on our findings at the end of our country visits. We had a report back to them after our emerging findings discussion amongst the commissioners. At emerging findings meetings, we got a very positive response from DFID. They thought that we had a great agenda; we had highlighted some important issues and made some important recommendations. Our sense was that they were really buying into the conclusions from our report.

When you look at the way this is articulated in the management response, I struggle to be clear whether there was complete acceptance of our recommendations. It was not clear whether there was acceptance but saying, “Actually, we have already done that now.”  Had we done it before or had we just done it since we got the report? I do not know whether it set out to deliberately avoid answering the big questions but, to my mind, it does not answer the questions. Are the recommendations accepted? What are we going to do about it and by when? We really hope to get those responses from DFID. We have a learning session with DFID lined up in a couple of weeks’ time, and we will certainly be wanting to debate those issues at that meeting.

Q22   Chair: Can I therefore ask if you will be going back to DFID in writing, with a point‑by‑point request for responses to your report, so that we can have something further from DFID and from yourselves?

Dr Evans: We have done already.

Chair: Thank you. Thank you very much for coming to see us today.

Examination of Witnesses

Witnesses: Nick Dyer, Director General for Policy and Global Programmes, DFID, Guy Howard, WASH Policy Team Leader, DFID, and Professor Charlotte Watts, Chief Scientific Adviser, DFID, gave evidence.

Q23   Chair: Thank you very much for coming to see us today. I think you have heard our previous inquiry session, so hopefully that will be helpful in terms of our direction of questioning for you. Could you first introduce yourselves?

Guy Howard: I am Guy Howard. I lead the WASH policy team in DFID.

Nick Dyer: I am Nick Dyer, Director General for Policy and Global Programmes.

Professor Watts: Good morning. I am Charlotte Watts. I am the Chief Scientific Adviser to DFID.

Q24   Chair: Good morning and thank you for coming. Turning directly to the amber/red rating for sustainability, you have heard reference in the earlier session to a number of concerns raised in the report about this and indeed the National Audit Office’s review in 2003, when they also criticised a lack of evidence and a lack of focus on sustainability. Why has work on sustainability not improved over the last 13 years?

Nick Dyer: I will turn to Guy and ask him to answer that in a minute, but can I just start by making a general comment, which is on the report, which we very much welcome, particularly the conclusion that we are making a significant contribution to extending WASH access? To us, that demonstrates that we are making a difference to millions of people’s lives. We should just record that. We are also particularly pleased to see the recognition that the results claim is based on credible data. That is a very good basis on which to start the conversation. Guy, do you start answering on sustainability?

Guy Howard: Yes, thank you. What came through in the report was a very good challenge to us. It poses the right questions to DFID to say whether we are doing enough around sustainability. Picking up also on the report and what you have heard from the ICAI commissioners, what is clear is that we have more to do to make sure that we are demonstrating what we are doing on sustainability, make sure that we are both recording that systematically and, more importantly, make sure that we are designing programmes that genuinely have sustainability at their core. In all our programmes, sustainability is one of the key functions of development of the business case and is the key thing that will go through quality assurance. I have done a lot of quality assurance of WASH business cases and it is the big question that you will ask yourself and ask the programmes to demonstrate how these results will be sustained.

              What we have not demonstrated sufficiently well is making sure that we are pulling out exactly what we are doing on system-building and demonstrating how that investment in system-building will ultimately result in sustainability. What we also need to do is to make sure we go back and check, when we have made big commitments on results, to look back after some period of time to see how many people have sustained access. That is what we plan to do. It is something we have on our plan, in fact, before the review was undertaken. In 2018, we will commission a survey, which will be done by an independent group, to take a representative sample of those programmes where we have delivered results to make sure we have a very clear idea of how many of those people we claimed got access, under the last results period, have sustained that access.

Q25   Chair: Thank you. Can I just probe you a bit further, as to how you are proposing, going forward, to improve in a systematic way measuring and showing sustainability, so that you can confront the challenges in this report that you are not approaching it systematically now? For example, where there perhaps should be engagement with national Governments that is not happening systematically. There is no monitoring of sustainability beyond the life of programmes in a systematic way and you have fallen behind on the donors. That is quite a concerning comment, after so many hundreds of millions of pounds have been spent and over 13 years and more of a very key programme.

Guy Howard: Let me take the first part of that, first of all. In terms of how we are going to do this systematically, we have working on our monitoring and reporting systems and we now require all offices that are funding WASH programmes to include, on an annual basis, an assessment of how much progress they feel that programme is making ensuring the results are sustainable. At the moment, we are allowing that to be a more qualitative response. In June next year, we will be reviewing how far we have got and, if we do not feel that we are embedding this strongly enough in terms of reporting, we will be more directive and give more directive, quantitative indicators that are being looked at.

Of course, you have to recognise that, in different countries and in different contexts, this will work in different ways. In Ethiopia, for instance, we are investing a lot in monitoring and evaluation within the government systems. We have done that previously in programmes that I have been involved with in Bangladesh, and we will also be doing it in countries like Sierra Leone moving forward. What we need to do is make sure that we are capturing that data effectively, we are bringing it together, we are synthesising it and we are benchmarking it. Next year, I would expect us to be in a position to say, “Here are the minimum levels of standards that we expect”.

              Your second point about whether we have fallen behind other donors is an interesting question. We have had a number of conversations with our Dutch colleagues in particular around this. A lot of what the Dutch do under the sustainability checks is right, and it is the stuff that we would also do, looking at the institutional set‑up and at financial and operational aspects. In fact, we are stronger on environmental sustainability than they are and we have a much stronger narrative on making sure that our water projects are underpinned by knowledge of water resources, their ability to serve people in the future and withstand future climate change.

The big question has been around the sustainability checks and almost the 10‑year clause, which is often used in the Dutch programme. Philosophically, I have a problem with that 10‑year clause, which I have pointed out to the Dutch before. It is that that assumes that, if something fails after 11 years, that is okay, because the sustainability has to be assured for 10 years. Where we want to go is to make sure that what we are recording and measuring is genuine sustainability. These are services that will continue to be delivered to people accessing them in perpetuity, essentially to everyone, everywhere, all the time.

Chair: Stephen might want to pick up on the timing.

Q26   Stephen Twigg: First of all, can you say more about the survey that you said is going to happen in 2018? What will that consist of?

Guy Howard: What we will be doing is, through a competitive process, we are going to commission a third party that will be specifically asked to come forward with proposals to survey right across our portfolio, so all the programmes that contributed to results between 2011 and 2015, and to design a survey methodology that will convince us, and has sufficient power and confidence to generate representative results. We will then expect that to be undertaken. My guess is that that survey will probably take the first six months of 2018, so a report is likely at the end of 2018 to early 2019, because there will need to be a period of analysis. In my experience, if you are going to do these surveys, you want the more sophisticated analysis to be done.

I will be honest: I have not yet written the terms of reference for that survey. That is something that is underway. What we will want to look at through that is not just functional status, but it will be to look at what those programmes have done, in terms of building national and local capacity to sustain access to services. Importantly, what have we done to build responsiveness in those systems when they find out that access has been lost? The issue about sustainability is reactivating access, not just saying that we have lost access.

We will look at the institutional side, but will also look at whether we have done enough to mobilise sufficient domestic resources to invest in water and sanitation to make sure that they can be sustained. Have programmes genuinely done enough to think about the environmental influences on sustainability? Are we confident that a borehole will last for more than five years?

Q27   Stephen Twigg: Is there any reason the survey could not be done sooner than 2018?

Guy Howard: We could do it sooner than 2018. We have always worked on that timeframe, because we wanted to allow sufficient time to make sure that we got a realistic picture of sustainability. There is a risk that, if you do it too soon, some of your services would only have been put in within the last six to twelve months, so therefore are probably more likely to be sustained. The idea was to give enough time to allow that first failure point to happen and then to go back and say, “If failure happened, did people reactivate the service?”  That, in a sense, is the key question you are trying to look at.

Professor Watts: I just wanted to comment on the research component of this. Clearly we all have the ambition of sustainability, but we do not have all the answers for how to achieve that. In the research portfolio, some of the areas that we are looking at are, for example, how we change social norms and how we create that local demand so that, for example, people will want to wash their hands using clean water. There is some research we have been doing to say how we involve the media and how we involve social marketing expertise to create a demand for clean water. There is a campaign called Choose Soap, for example, which Unilever has been implementing and we have been evaluating to say if we can support the demand side to this, because we think that could be quite important in terms of sustainability.

              The other one relevant area we have been looking at is the role of the private sector. Our discussions have very much been about the role of Government and political commitment in funding, but we also think there is a potential opportunity for the private sector to be involved. For example, we have research that has taken a smart pump, which sends out a signal when it is not working. We have been trialling an intervention in Kenya of people buying into access of this pump service, showing it is creating a mechanism that the private sector can benefit from, which ensures continuous functioning pumps and that people have access to that. That is something looking very promising, which we are hoping might be incorporated into the new water Bill in Kenya, in due course.

Nick Dyer: The smart pumps are really effective. The downtime of pumps has gone from an average of 44 days down to four days, because people are getting real‑time information on when the pump has stopped working, and then can send out an engineer to fix it.

Q28   Stephen Twigg: Can I take us to the issue of programme length? In your management response, you say that WASH support is delivered over a longer period of time through multiple sequential projects, but ICAI’s point, as I understand it, is that trying to include a two‑year sustainability phase in a five‑year programme risks compressing the time that is available for delivery. Is there not an issue here about trying to do things too quickly?

Guy Howard: There is a question about overall programme lengths. Obviously the longer a programme is the better it will be. Our point about the sequential nature of a lot of our programmes is that a lot of the investment in sustainability and building local capacity has to be seen in the context of repeated investments. A good example, for instance, is a big programme in Zambia that focused on sanitation and hygiene. We came to the end of the delivery phase. We then extended the programme by a further two years in order to work with the Government of Zambia to build their capacity.

              On the specific example that was quoted of Mozambique, there is a little bit of a misunderstanding about the nature of that programme. In a sense, we did not compress delivery into three years by having a two‑year sustainability programme out of a five‑year programme. The three years for the output delivery were dictated by the time between approval of the project and the deadline for meeting the results target. What we then did was, accepting that three years was too short, we deliberately designed that programme to have a further two years on top that would not focus on delivering first‑time access to people, but would focus solely on embedding sustainability.

What is interesting about that programme is that it also uses Payment by Results within that outcome phase, so suppliers actually have to demonstrate to us that they are improving and ensuring the sustainability of those services. If they cannot do that to the satisfaction of a third party, they do not get paid.

Q29   Stephen Twigg: Will you look at extending programme length? Is it on your agenda to look at that? I accept your point that things vary between different countries with different situations but, as a general rule, would you look at extending programme length?

Guy Howard: As a general rule, I can only talk in terms of the WASH policy.

Stephen Twigg: Sorry, I meant a general rule regarding WASH policy.

Guy Howard: There is a policy question for DFID there, which I am not the person best placed to answer. In terms of WASH programming, we would always be encouraging people to make sure they are designing programmes of sufficient length that will provide confidence that there will be sufficient investment to ensure sustainability. If people come forward with two‑year programmes, when those go through a quality assurance process, the immediate challenge back will be, “We do not believe that, in two years, you will be able to sustain these results. You need to think about how you will do that”. We need to do that.

We also need to look at how we design our programmes. Our programmes increasingly need to focus more on building that capacity, so less on delivering taps and toilets in the first instance and more on ensuring we are creating the institutions, public and private, which will ensure that those services can be sustained. That is a change that we will see over the coming years. It will take some time to play out, but that is certainly where DFID Nigeria, for instance, will go on their WASH programme.

Q30   Mrs Grant: Some stakeholders have expressed concern that Payment by Results does not always provide the right incentives to suppliers to focus on sustainability and sustainable results. How are you making sure that suppliers are incentivised to deliver sustainability and, once you have answered that, could we also have an indication of what is happening with the Mozambique pilot?

Guy Howard: In terms of the incentives, again it is a very good question. The context of this has to be that Payment by Results is very new in the WASH sector. This is not a modality that has been widely used, and so we have all been trying to find our way through this. I do not think we would say that we have a perfect answer.

What we have done through our existing programme is, through building in an outcome phase to the WASH Results Programme, which is the Payment by Results programme, and by linking payments to suppliers to demonstration of sustainability and sustained outcome, which is verified by a third party, that is creating a lot of the right incentives for suppliers to focus on that as an issue, because there are hard financial penalties if you get it wrong. Having just gone through a set of negotiations with people over the last two weeks, people really feel this, particularly in the NGO world. They do not have lots of assets to play with.

              There will be more that we need to do. We need to learn from going through this programme and we need to look at whether we have got it right and whether we need to improve what we do. We will do that in dialogue with our suppliers to make sure we do get it right.

Specifically on Mozambique, it is one of those difficult situations and I fear my answer may not fully satisfy you. It is a little bit too early to say, because we are just going into the outcome phase. In two years’ time, I can come back to you and give you a very definitive answer. What I would say is that what we have seen so far, based on the independent advice we have had from our third‑party verifiers, is that what is being proposed is good, robust and anticipated to lead to sustainable outcomes, but the truth, the real testing of that, will be in two years’ time when we look back at the end of the programme.

Q31   Mrs Grant: Do you both agree with that?

Professor Watts: I just wanted to add on the Payment by Results, because it is a relatively new mechanism across a number of programmes in DFID. As well as the learning that is happening within WASH, we are also about to do a learning exercise across a number of Payment by Results initiatives, including in girls’ education, to try to understand what value these have, what ways there are to set up mechanisms in an appropriate way and what we can learn from our existing programmes.

Q32   Jeremy Lefroy: There seems to have been a slight discrepancy in terms of the numbers of people reached and whether that was based on actual counts or whether it was based on population assumptions. I wonder if you could just comment on that and whether DFID has a clear methodology for counting people reached, whether it is based on physical counts or assumptions from demographics.

Nick Dyer: I will answer, just to give Guy a break, and then I will ask him to elaborate on this in a minute. There has been some confusion in the conversation that we have been having on this. With water supply, we are generally using the well-established methodology that is used within the sector, in terms of looking at populations and taking the average reach within the site of a borehole or a water point. The confusion arises that, in sanitation, there has been a different approach in some cases, where there has been more direct counting, particularly for communities that say they are now open defecation free, then you can count the community. Guy, do you want to elaborate on that?

Guy Howard: Absolutely. Thank you, Nick. On water, the ICAI report is absolutely right: we do make that assumption and that accounts for about 22 million from our results returns, so about a third of the results. They recognise and we recognise that it is an imperfect methodology, but one of the issues is that access changes so hugely over time that any estimate is only an estimate at that point in time, and things can change.

On sanitation, the confusion probably comes around a slightly different interpretation. I went back to our methodology notes, and I think we have a job of work to do to make sure that we are clearer and that there is no ambiguity in that. What we expect people to do with sanitation programmes, if for instance they are doing a community‑led total sanitation programme that will result in communities being declared open defecation free, is estimate the number of households that have toilets before the start of a programme. Once the community is declared open defecation free, and therefore they have access to toilets and are using them, they must make sure that what results are counted take away that group of people who already had access to a toilet. We have challenged, for instance, from a centrally management programme, UNICEF and our WASH suppliers. We demand evidence that they have done that.

              There is obviously an assumption about household size. In some programmes, people count everybody in the community. In other programmes people may assume an average population household size but, in general, it is a more direct count of the number of people who now have access to sanitation. I would take the point that we need to be clearer in our methodology and we need to make sure that the programmes are genuinely reporting that, including to review teams when they are coming through.

Q33   Chair: Can I just ask if this was raised in the fact‑checking exercise?

Guy Howard: It was raised. We did raise it. Indeed, it was point 62 in the spreadsheet. However, looking back at that spreadsheet, perhaps we were less clear than we should have been and I apologise for that. That is something that is a good lesson learned for us. We should have been much clearer in our view.

Q34   Jeremy Lefroy: Thank you for that. Just as a follow‑up, do you also look at some behavioural change? Clearly if you are talking about open defecation, if latrines are being provided but people are not using them, then that makes a big difference.

Guy Howard: We do. When we are thinking about sanitation, if we are using the model of looking at communities that are open defecation free, that is looking at behaviour. Normally, what we also do is not just make sure that the community is free of faeces, but also follow that up with surveys to make sure that that does not just mean that everybody is just going round the corner or going two fields away but are genuinely using latrines.

Hand‑washing, which is the other big behaviour change, remains a challenge, as Charlotte has already flagged. There are proxies that we can use, around looking at the presence of soap and water within a toilet. However, without doing quite expensive structural observation studies, it can be quite difficult to prove that people are washing their hands all the time. There have been interesting proxies that have been developed previously around fingertip-rinsing, which again can help you understand whether people are washing their hands. The problem is that all these things cost quite a lot of money to do and you need to do it repeatedly.

One of the issues around hand‑washing is that probably no one, anywhere in the world, washes their hands every time that they should do. You only have to look at the surveys that are done in the UK to look at the levels of hand‑washing. Globally, it is estimated that only about 19% of the population washes their hands when they should do, so achieving sustained behaviour change on hand‑washing is a real challenge. As Charlotte said, this is what we are looking at through research. We are trying to find other ways of trying to incentivise people to change those behaviours, but we recognise that it is a long‑term challenge.

Q35   Chair: Can I turn now to data on impact, the difference made in the lives of beneficiaries rather than the numbers reached? The report, at page 17—and I will just quote it, if I may—says that, “Examples … indicate that wider development impacts are being achieved through DFID’s WASH programming. However, from the evidence available, we cannot reach conclusions as to where and to what extent these impacts are occurring. Impact data is not routinely available at the programme level. Nor is it aggregated at either the international or the country level. Around half of the programmes in our sample planned to conduct an external evaluation of some type, but many are not set up to generate the evidence that would allow impact to be measured.”  A key recommendation of the report is that “DFID should improve the measurement and reporting of the development impact of its WASH programmes, particularly for vulnerable groups”. I would like to ask Mr Dyer how you are going to improve your approach as a result of this recommendation.

Nick Dyer: I would start by saying that there are development interventions that we know work. If you vaccinate a child from polio, you know you will protect them from polio. You do not actually need to measure the polio protection; you measure whether you are reaching the number of children and whether you have sufficient reach. That is the key indicator for vaccination. We know that, if you use an insecticide‑treated bed net, you will protect a child from malaria. Again, the issue there is not testing whether there is protection from malaria and that impact. You just need to test whether you are reaching people with bed nets and that they are using them. There are circumstances where reach is the key indicator.

              The question in WASH is whether that is sufficient. Now, we know that well designed and implemented interventions in WASH reduce diarrhoea. If you are after a public health intervention, which we are with WASH, we know there is going to be a public health impact if you get the reach you are looking for. Reach is an important part of a proportionate way of measuring your impact, because you know this works.

It is a good challenge as to whether there are other development impacts in WASH that we should be seeking to measure better. There are a lot of claims out there that the evidence does not back up. People claim that toilets in schools prevent sexual violence against girls. Actually, there is no evidence either way of whether that works or whether it does not. People make claims for deworming improving school access and school improvement. Again, the evidence is really contested on that. People make claims on the evidence for WASH and nutrition. Now, that is better; there is medium‑strength evidence that WASH does improve nutrition. We are definitely cautious about what we claim for WASH, so we recognise that this is an important issue.

              Our programmes in DRC and Ethiopia are going to be doing evaluations, which are going to look at the health impacts of WASH. We are commissioning a series of evaluations that will be looking at measuring the impact of WASH on nutrition, health and education. These will be happening over the next two to three years. The Chief Commissioner is right; these are really expensive to do. We have to be very selective in what we do. I have asked the team internally to go back and have a look at our guidance to our teams in the WASH area, in terms of whether we are handling and recognising development impact sufficiently, in terms of how they design their projects and programmes. We recognise it. I would attempt to have a bit of caution in this, but we are putting investments in evaluation and we are going to be looking at our guidance again.

Q36   Chair: I think all of us would agree that there certainly is some wider impact and the report acknowledges that. It is how to ensure that you can implement what the report calls “convincing methodology” and obtain consistent data to enable you to measure a return on investment and identify inefficient programmes. That is really the difference between looking at a project and saying, “Well, yes, it’s having some impact”, and forensically scrutinising it. Are you going to change your approach to ensure that you can do that, so that you can compare efficient and inefficient programmes, or are you saying that you disagree with the National Audit Office when it said that it is unlikely sustainability will be achieved if not monitored?

Nick Dyer: It is certainly not for me to disagree with the National Audit Office. For me, the absolute must‑have in WASH projects is that there is a public health impact. We have to be absolutely sure that we are getting a public health impact. On top of that, we want to be cognisant of the additional development impacts that could occur. Where we have education programmes that are investing in school toilets, then we need to do a better job in getting our education advisers to understand that they need to build that assessment into their programmes, in terms of what the impact of those school toilets is on educational access.

We are now doing a series of nutrition interventions looking at the links between WASH and nutrition. Yes, there are ways that we are improving the internal guidance to our teams to say to them that, when they are using WASH in these circumstances, they should be assessing those development impacts. Guy’s teams are not necessarily running all the WASH interventions we are doing in DFID, because some of them are coming through other sectors like education.

Chair: We might be interested to see a further, more detailed response if ICAI raises this with you again.

Guy Howard: Can I just add that, in terms of understanding inefficiencies to the programme and linked to the value-for-money areas, one thing we have done since the review is to produce a guidance note for our country teams working on WASH? We have shared that; that was disseminated in May, both directly to those programme staff but also put on our internal website. We will be monitoring the use of that guidance and making sure that we continue to work on it, refine it and improve it. We will use that as a way of starting to create a database that allows people to benchmark performance.

I feel that we are getting into a better place. The criticism is a reasonable one that we are not doing it systematically enough, but I would say that I am not aware of any sector or donor that has managed to crack this particular nut. Even the World Bank, which you would assume has the people that should be leading the way on developing lots of good methodologies for value-for-money assessment in WASH, is struggling at times to articulate that. It is a challenging area.

Q37   Chair: Can I just ask a specific question? You set yourselves a laudable target of reaching another 60 million over the next few years, with the WASH project. Will changes to the definition of access to basic water, now requiring a round trip to a water point to be less than half an hour, cause you to perhaps review the areas that you will be able to access and possibly even deprioritise remoter areas? How are you going to approach this newly phrased and worded target?

Guy Howard: This issue about the change in definition is an interesting one. Actually, it was there in 2002 in the original methodology for the joint monitoring programme. I remember it well, because I reviewed that at the time. It has gone in and out of their methodologies all the way. What is clear is that all the surveys that they use ask questions specifically about the time it takes to collect water, and they use that in their analysis. Whether it is a genuinely new level of access is questionable.

The short answer to the main question is absolutely not: I do not think DFID should use any tightening of any definition of access for us to say that we will not go in and target particularly marginalised or vulnerable groups from remote areas. That is precisely what aid should do, because others struggle to do it, so I do not see it having any major impact. What would have an impact is if we accepted the proposal that what the SDG really means is water supplied to the house, for which there is a very good health, social and economic justification. That would create a problem for everybody to achieve, but that is not on the table.

Nick Dyer: Can I just add to that? The other implied question, when we look at this, is whether the costs of reaching more marginalised people will mean that we are going to spend more money in this area. It is more expensive to reach people. The reality is that, for an organisation like DFID, which is working in many fragile and conflict‑affected states and dealing with very marginalised groups, you will find that, in some cases, our costs will go up, because reaching the last mile and the hard to reach is expensive.

Q38   Jeremy Lefroy: You will have heard the questions that we had earlier about value for money and I wondered what the current situation is. Perhaps I can ask Guy, first of all, in terms of where DFID is going. It seems to me that, as the Chief Commissioner was saying, perhaps there is a lot of practice there that is not put down and perhaps things are going on that we do not know about. It seems to me that this is a fundamental point that needs to be in place and does not appear to be in place.

Guy Howard: We would accept that we have quite a lot more to do around this area. The review referred to the research that we commissioned, which has helped us get to a point where we have been able to develop guidance, as I have said, on how to do value for money and to share that with our advisers and programme managers working on WASH programmes.

Moving forward, we are very clear that we need to get a much stronger database and a much stronger set of case studies and examples, so we will be reaching out across our network and across the portfolio to make sure that we are drawing out those case studies and that we maintain an active website that gives people examples of how to undertake value-for-money analyses and the kind of data that comes back out of those value-for-money analyses, so that we have benchmarks that people can start to look at. They can genuinely start to say, “In my programme, delivering water is costing this. How does it compare to a country neighbouring me that has a very similar context, hydro‑geology and all the rest of it, and what am I getting from different partners? Am I better going with UNICEF or am I better going with NGOs?”  That is something that is very much work in progress and, by June next year, I want us to be in a position where we have a lot more of that in a central repository, being shared across the organisation.

Q39   Jeremy Lefroy: That is encouraging to hear. If I can pick up on that, you mentioned UNICEF, which as far as I understand is the implementer in something like 60% of DFID’s programming, but they do not appear to be subject to market testing. Why is that?

Guy Howard: Again, that is a very good challenge. Part of the reason is that, as a UN agency, UNICEF often will not go into direct competition, although it has done in some countries. I know in Sudan they did go through this process with UNICEF. UNICEF is an important partner. They delivered about 60% of our results. From what we have seen from the financial data, that accounts for about 43% of our spend on WASH. At a very crude level, they are obviously being able to deliver quite good value for money, but the challenge is the right one to make sure that we are properly shaping the market. That was a lot of the reason for doing the WASH results programme. Through that process, we have started to develop and build new consortiums and new alternatives to UNICEF. We are very actively encouraging all of our programmes in country offices to genuinely look at the choices that are there.

What I would say is that part of the business case process on any programme is to do an appraisal of options. Even if UNICEF is not formally tested in the marketplace, it is absolutely the requirement that people have looked at them in addition to the other options and done a costed appraisal about which offers the best value-for-money approach, which is a standard method of all our business cases.

Nick Dyer: Can I just add to that? As we have said previously in this Committee and in the broader IDC, in DFID now we are taking a much more commercial approach to our programming and investing in our commercial capability within the organisation. My expectation is that we will be more regularly testing what the market is and looking to build the markets that currently do not exist. Whereas UNICEF may be the only reasonable supplier in a particular country, we would look at options in terms of how we could build that market so that others could come into it, so that we are not just relying on UNICEF the whole time. That is something across the DFID programmes that we are endeavouring to do more.

Q40   Jeremy Lefroy: If I could just come back to the point I was asking before in terms of demand, compared to other agencies DFID’s WASH programme substantially works in the more difficult‑too‑reach areas, which I welcome. In those more difficult‑to‑reach areas, demand is quite clearly key to ensuring sustainability. If people are really asking for something, they will ensure that it works and continues to work. How do you respond to that, as opposed, say, to putting in a major water supply system to a city, which really has to be done at regional or even national Government level? Once it is in there, people will tend to use it.

Guy Howard: That is a very good question. The answer is twofold. At the macro level, the country level, this is around a discussion with Government, both national and local, and other sector partners, about looking at where needs are. That is largely driven by data that show which are the areas and communities where people lack this basic service. To some extent, there is always a supply‑driven process to this, because the benefits of water and sanitation as a public health intervention are not just accrued to the individual. It is about the wider public benefit. We would always be insisting our programmes are focusing on those areas that do not have access, rather than just trying to incrementally improve the service for people who already do have access.

              The second part of it is at the more programmatic level, where particularly on sanitation, but also to some extent water, we would ensure that all our programmes follow a demand‑based approach. There has to be a demand from communities for their services, in order for them to be sustained. Our experience on water is always very simple. It is very straightforward. It is nearly always about the first thing someone will ask. It is always more challenging with sanitation, which is why, for a lot of our sanitation programmes, there is a process where that demand has to be created. We support the community‑led total sanitation programmes in countries like Zambia or Mozambique. You have community volunteers who go out into the communities and work with communities to create that demand. You are absolutely right and we would absolutely insist that we are responding to demand because, if we are not, then sustainability is going to be very difficult to ensure.

Q41   Jeremy Lefroy: In the case of the report in Zimbabwe, where even after two years the programme was not functioning or only partially functioning, why would that have happened?

Guy Howard: That is an interesting question. It is slightly difficult, because I do not know the specifics of that particular programme. It was a school‑centred programme, which creates an additional set of dynamics over a community‑based programme, because communities’ expectations about who is responsible for pay and for operational maintenance will be very different. Certainly in Zimbabwe, while this has perhaps degraded over the last 10 or so years, if I think back to the mid‑1990s when I worked there a lot, it had a very strong tradition of very effective public management of water and sanitation services. There probably is still an expectation that people expect Governments to run services.

However, having said that, it is a good case in point. Clearly that programme has failed to ensure sustainability and we need to go back, look at and understand the reasons why. We need to make sure that we act on those and make sure that those mistakes are not made again in the future.

Q42   Chair: To just carry on with value for money for a moment, you heard our witnesses from ICAI express quite grave concerns about this, in fact saying that it was a close judgment as to whether they include a reference to value for money in their amber/red element of the report and that they were not able to draw conclusions about the overall value for money of the whole portfolio because of lack of information. They also indicate in their recommendations that there should be a sense of urgency about addressing this; DFID should act quickly. You talked about benchmarks. Can you give us some specific examples of what ICAI suggests ought to be credible benchmarks for measuring value for money? They also suggest there should be some more convincing methods for comparing costs. Can we just drill down a little bit and be really specific about how you are going to approach this, with this sense of urgency? Obviously you have had this report in draft for a few weeks now.

Nick Dyer: I have two responses to this challenge of value for money. One is whether, broadly as an organisation, we are giving sufficient attention to value for money. If it is not showing up in one sector, you have to ask the broader question as well. We have a lot of guidance within DFID on what we mean for value for money and how we measure value for money. We have expectations built into our business cases that people will look at value for money when they design a programme. I am satisfied that those have wide reach and understanding within DFID.

When it comes to a particular sector, this guidance note that Guy refers to, which we have just issued on WASH—and Guy can talk about the particular indicators—raises specific questions around how you treat an infrastructure investment; I forget who said it. You should be looking at the lifetime costs. There are challenges around looking at the capital and measuring the ongoing maintenance cost. There are questions around how you discount that as well, in terms of bringing it to a measure for comparing the costs over time, and there are particular challenges in terms of the specific technical interventions. Are there specific examples of the kinds of measures that we are asking people to be giving us now?

Guy Howard: We are looking at three levels. We are looking at people collecting data on unit costs, so how much it actually costs to get a toilet into a household or get a water supply into a community. What we are saying is that that needs to encompass all the costs of doing that. It is not just the purchase of the drilling rig or the hiring of the drilling rig; it is building in staff time and outreach. That is particularly important on the sanitation and hygiene components. It is actually relatively straightforward on water, to be honest, because water is a hard infrastructure intervention. Most of our sanitation interventions are not; they are behavioural. We are saying that people need to capture that.

We already have some benchmarks for that across a total of around six country programmes plus our two centrally managed programmes, where we are able to give indicative ranges of unit costs for each type of intervention and average costs. We are also looking at efficiency, which is really the cost per output, so the cost of someone getting access to that water supply. It is moving beyond just the physical costs of the infrastructure in saying, “That certainly needs many people, so what is the efficiency of that?”  Again, we have benchmarks on that across the same range of countries, with different types of implementing partners. Ultimately, we have measures around effectiveness, which are around sustained use of that water supply and how much it costs to achieve that.

The challenge remains a good one, in the sense that we do not have enough of this data. We need to get more of this data. We need to have a better sense of benchmarks. One of the things that has particularly come out of the work that we commissioned Oxford Policy Management to do for us was a clear sense that a lot of our implementing partners collected all the data you needed to do value-for-money analysis, but it was not brought together in a way that you could do that analysis, and so we are making a stronger push so that that is exactly what is going to happen. It is not just for DFID, but this is about us working with people like UNICEF to make sure they collect the data in a way that permits that.

              On that point, what has been very encouraging is that, last week at a strategic resource partners meeting with UNICEF, they have now committed that they will start to issue their own guidance on collecting value-for-money data. It is going to be embedded as part of their core indicators. That is a direct consequence of the last two to three years’ worth of work that we have done with them through a centrally managed programme, where we have insisted that they need to start demonstrating this more effectively.

Q43   Chair: Just very quickly going back, Mr Dyer, to the definition of value for money, in DFID’s document of its approach to value for money of July 2011, DFID says that it means maximising “the impact of each pound spent to improve poor people’s lives”. Do you agree?

Nick Dyer: The value-for-money indicators look at economy, efficiency and effectiveness, so yes.

Q44   Stephen Twigg: Can I take us to the issue of how the Department is going to improve its learning arising from this review? ICAI recommended that DFID improves how learning from WASH, including from its research programmes, is shared throughout the organisation. Can you tell us if you are developing a structured process for compiling and sharing lessons and in particular sharing knowledge from other related areas like health and education?

Professor Watts: Thank you. Shall I take this one? We are getting some things right when it comes to learning. For example, we do have a lot of communication and lesson learning within the WASH cadres. The challenge is how we move to exchange across our different sectoral specialists. We are thinking more broadly and are very involved as an organisation on how we support learning across our programmes. We have been looking quite closely at the different systems that we have in place and what incentives we have for people learning and sharing evidence, both the formal evidence that we generate through our research and also the more practical experience of running programmes. We are running training programmes across DFID at the moment to support staff to be better equipped to learn and ensure that they are able to access the evidence that they need. There are a number of initiatives that we are going to put in place this year and we will be assessing how well they are working in this particular sector and across nutrition and education in 2018.

Q45   Stephen Twigg: You mentioned training programmes, which sounds positive and encouraging. Reading the management response, I felt there is a risk that we could get lost in all the documents. It is great that every document has the learning in it but, unless people are reading all of them, the risk is that things just get lost. Is there something beyond the training programmes that ensures a proactive response?

Professor Watts: The training is quite proactive, because it is very much trying to say how we change the way that people work in their day-to-day business. Designing and developing programmes influences their work overall. It is not as if it is a one‑off thing, but it influences what they then do from day to day. We are supporting in this area a community of practice, so that is the other thing that we do actively. We can monitor and report back on that in terms of the level of activity and the numbers at community and practice events. That is one way we can monitor and report on that.

Nick Dyer: This has rightly been a regular challenge from ICAI. We have tried things in the past that we have concluded have not worked, so we have actually invested quite a bit of our time in putting together collections of information that people can resource and go to, but they do not. They do not access it. People access information in different ways, so we are trying to move away from collections to think about how we can connect people in a better way. These communities of practice have been more targeted in terms of the people you are trying to reach to see whether that can work better.

We are trying to systemise this. We are looking technically at our internal tools, in terms of how we can do record searches, because they are not as good as they should be at the moment. We are trying to improve that. We are systematically trying to think through how to improve this. I would expect and hope that ICAI keep challenging us on this, because this is hard and we are trying to get better.

Professor Watts: I just want to add that, in terms of research and use of research, we are having quite a lot of success there. As part of the contracts that we give to our WASH research organisations, they have to develop very accessible evidence syntheses on a range of topics. We have seen that feeding quite directly into a WASH programme, so we do have some successes around use of evidence and generation of evidence, but there are also ways that we can improve.

Q46   Chair: The final question will hopefully elicit a very short answer. Your management report this month was somewhat surprising, partly because it is relatively short. Considering the fact that you acknowledge that this is a very valuable programme, and considering the extent of the report and specific concerns and recommendations within it, in fact, your response says very little to confirm what DFID will additionally do to improve areas where ICAI has expressed concerns. Our question is: will you commit to providing a detailed point‑by‑point response to ICAI’s recommendations and issues of concern? Will you also respond to a separate letter from this Committee?

Nick Dyer: We have had various conversations in this Committee about the old format, which we both found led to some misunderstandings. We found it a bit inflexible and quite burdensome. We were hoping that this management response would give us an opportunity to elaborate more on the findings. I think it is probably fair to say that this has not been universally well received, in terms of what we provided.

I would make the point that the format and the content of our management response is agreed by the Secretary of State. It is the Secretary of State who decides how we respond. We have received the letter from ICAI on the management response. I know the Permanent Secretary spoke to the Chief Commissioner about it and I know that the Secretary of State is aware of it as well, so it is on her radar. We are happy to go away and have a look again at our management response to provide something that people are happier with. I cannot commit that this will be immediate, given everything that is going on and that this has to go through the Secretary of State, or that it will necessarily be that quick, but I can give you my assurance, Chair, that we will be going away and having another look at this.

Q47   Chair: Thank you. On the question regarding a separate letter from our Committee, would you also be willing to look at that?

Nick Dyer: Yes, we would be very happy to.

Chair: Thank you very much for coming today.

              Oral evidence: Sub-Committee on the work of ICAI, HC 102                            2