International Development Sub-Committee on the Work of the Independent Commission for Aid Impact
Oral evidence: ICAI’s review on assessing DFID’s results in improving maternal health, HC 1733
Wednesday 19 December 2018
Ordered by the House of Commons to be published on 19 December 2018.
Members present: Paul Scully (Chair); Richard Burden; Chris Law; Henry Smith; Stephen Twigg.
Questions 1 - 41
Witnesses
I: Dr Alison Evans, Chief Commissioner; Emma Back, Team Leader, Independent Commission for Aid Impact; Richard Clarke, Director General for Policy, Research and Humanitarian, Department for International Development, Independent Commission for Aid Impact; Will Niblett, Head of Sexual and Reproductive Health and Rights Team, Department for International Development.
Examination of witnesses
Witnesses: Dr Alison Evans, Emma Back, Richard Clarke and Will Niblett.
Q1 Chair: Thank you very much for coming, everybody. We have a good report to get through today, so I will, as ever, ask whether you could start us off, if you want to make a brief comment. Please do be brief, because I am sure we will tackle most of the issues as we go through the questions anyway, but feel free. Can I start with you, Alison? Thank you very much for your last appearance before us and all of your work to date. Looking at the methodology, as ever, it included case studies of Malawi and DRC. On what basis did you choose those countries?
Dr Evans: Thank you very much, Chair. It has been a pleasure working with all of you, too, and I wish you all the best in your continued work with ICAI. Let me take up your offer and say a few things about context and why we looked at this particular area of DFID’s work, and then address with Emma the specifics around the methodology.
Around the world, something like 300,000 women a year die in pregnancy and childbirth. That is around 830 women a day, and 99% of those deaths occur in developing countries. Well over two-thirds of those occur in sub-Saharan Africa. It is one of the principal killers of young women and girls between the ages of 15 and 19, so this is a hugely significant area and a major area of concern. Reducing maternal death was also a key priority area in the millennium development goals agenda and one of the few targets that were not met at the end of 2015.
DFID has had a consistent focus on the area of maternal health. Going back to the first White Paper in 1997, there was a focus on reducing maternal mortality. The first maternal health strategy was in 2004, so this is a longstanding area of concern for the Department. We therefore decided that, as an area of relatively mature programming and longstanding commitment, we wanted to undertake an impact review of the key results claimed in terms of maternal lives saved in this area, during the principal period of the results framework, which is 2011 to 2015.
We took the overall results claim that DFID produced in its 2015-16 annual report, and drilled down through it to try to understand where those results were coming from and how we could sample programmes accordingly, to capture the programmes that were contributing most to that overall results claim. We did that by looking at a set of centrally managed programmes the Department has. It identified something like 15 overall and we sampled the eight that were making the largest contribution to that results claim.
We also sampled two specific countries, where we wanted to follow the results all the way down to the field level to understand the relationship between what we could see on the ground and these overall results claims. I will ask Emma to elaborate our country choice criteria, but in total our sample of programmes contributed over a third, roughly 36%, of DFID’s overall results claim. We were able to sample just over a third of the total contribution to that big result claim in 2015. Do you want to say something about case studies?
Emma Back: We shortlisted five countries as potential case studies and discussed those with DFID, at both headquarters and the country offices concerned. In the end, we focused on DRC and Malawi because of their contribution to the results claims, as Alison has just outlined. Malawi was the largest contributor overall to DFID’s results claim and the largest from family planning interventions specifically, whereas DRC was the largest contributor from healthcare interventions, so initiatives focused on maternal, neonatal and child health.
In addition, both countries had broad bilateral portfolios so, as well as being able to look at family planning, maternal health programming and health system strengthening programming, we were able to look at other health sector programmes and take a quick look at programmes beyond the health sector, to assess the degree to which those were contributing to maternal health outcomes. Finally, we were also able to look at some of the centrally managed programmes that had a presence in DRC and Malawi. Based on the programme portfolio, we thought these were strong case studies. They are both countries where DFID is likely to remain for some time and, therefore, they have a strong lesson-learning potential going forwards, and they are countries where maternal mortality remains very high, so significant progress is going to be needed right the way through to 2030 and beyond, in line with the sustainable development goals.
Q2 Chair: Richard, I know you initially estimated that you had saved 103,000 maternal lives between 2011 and 2015, but the internal analysis reduced that to 80,100. You indicated that the final estimates would be published in 2017, but I do not think they were. Can you tell us why not?
Richard Clarke: Thank you, Mr Chairman. I would also like to take up your offer, if that is all right, by saying a few words at the outset. I will start by expressing my thanks to the Committee for the opportunity to come along today, and to ICAI and, if I may, particularly to Alison on her last appearance. We have really enjoyed working with Alison and the team over the last couple of years and we are grateful to ICAI for its measured and helpful recommendations on this work.
We have always been clear that this is a very complex area. It is one of the most challenging areas in which we work, and we work on it in some of the most challenging countries in which we operate. Maternal health is an issue that goes to the heart of our development work and, as Alison said, the statistics remain stark in this area, so our vision is bold and, we believe, rightly so. Our ambition in this area has never faltered and it continues to be a central part of our programming, both bilaterally and through major multilateral elements of our work.
We were very pleased to receive ICAI’s timely report. We place huge emphasis on monitoring, learning and development in everything DFID does. To have a fresh perspective on this from ICAI is an opportunity we are determined to seize and, I will say at the outset, we have accepted all five of the recommendations.
You asked, Mr Chairman, why we did not publish the revised results at the point we said we would. That was an oversight. It was always our intention to do so. As you say, and I am sure we will come on to this in more detail, we use a range of different approaches in different countries to estimate the impact that our programmes have had. In both Malawi and DRC, we used forms of modelling to assess whether our projects have had the outcomes we would expect. In two of our countries, we received revised analysis that brought our overall numbers down from the 103,000 figure you say to just over 80,000. That was still well in excess of the target we had set as part of the 2011-to-2015 period.
It had always been our intention to publish it as part of the annual report. We should have done so; it was an oversight that we did not. During the conversations with ICAI in preparation for this report, it became clear that we had not done so. As soon as feasible, we published that and two other elements that were in the same package, on 5 December, in a public report.
Q3 Chair: I have seen in the news DFID’s rebuttal to the media coverage around those figures, and one of the things you particularly pushed back on, against ICAI itself, was that you did not feel this was representative of your global work. How did you describe it, “after visiting DFID programmes in just two of the countries where we operate”? After what you have heard from Alison and Emma, do you have a response to why they were querying just two countries?
Richard Clarke: We were concerned that some of the reporting suggested an abstraction from those two country offices. We recognise that ICAI took a range of approaches to analysing the work that it did for this report, including both intensive field work, with full co-operation from the country offices locally in both Malawi and DRC, and conversations with the team. It clearly drew on a range of important resources. There are some important areas on which we respectfully disagree, in some of the conclusions, but we respect the approach that they have taken.
Q4 Chair: Alison, do you have any feedback on that?
Dr Evans: The thing to note here is that it is not at all unusual in an ICAI review for us to do a wide-ranging assessment of programmes that have a global or regional reach, then take a couple of case study countries and do something in depth. At no point in any of our reviews do our judgments rely only on those in-depth case studies. That is the case here, and it is the case whether we have green/amber or amber/red reviews. We took an exception to the suggestion that we had built our impact judgments simply on visiting a few facilities in two countries. That was clearly not the case.
Q5 Chair: ICAI was arguing that, in estimating the number of maternal lives saved through your family planning programmes, you were overestimating the effectiveness of long-lasting and permanent methods of contraception, predominantly taken by older women, but younger women account for a large proportion of maternal deaths. What are you doing to take account of ICAI’s findings in that regard?
Richard Clarke: I might say a couple of words and then ask Will to come in, if that is okay. The first thing to say is that we definitely regard contraception as part of a broader range of interventions that we have tried to take forward, over the last few years, and particularly during the period ICAI covered in this review. It is important to emphasise from our perspective that, when thinking about family planning and the range of potential interventions, there will always have to be an element of modelling. We have to make assumptions about how people are likely to behave and those are never going to be completely perfect. In this case, we made some assumptions that we think are reasonable in that context, but I will ask Will to say a bit about the specifics.
Will Niblett: Good morning. I am Will Niblett, from the sexual and reproductive health and rights team in DFID. As Richard said, these were modelled results and modelling inevitably has limitations and relies on assumptions. The particular case that ICAI was talking about was in Malawi, where we had to use a proxy. Ideally, we would not have had to use that proxy, but we did so because it was the best available data. We accept that using this proxy in Malawi means that the modelling assumptions were at the less conservative end of the methodology spectrum that we use, but we maintain that they were still credible.
Q6 Chair: What do you mean by “proxy”?
Will Niblett: We would normally have used the contraceptive prevalence range to work this out but, in this case, we had to use couple years of protection. It is quite a technical issue, but that is what you were picking us up on in that case. We used it just in a couple of countries.
Dr Evans: It might be worth getting Emma to elaborate a little on why that is an issue, without going down too much of a technical rabbit hole. In response to Will, yes, we raised concerns about the modelled estimates for Malawi and looked at them in great detail, but we subsequently found the same proxies had been used in a number of other countries, which was acknowledged. That then raised questions about the robustness of the estimates in other countries. While we were not able to look at 100% of those estimates across all countries, we had reasons to be concerned that all those estimates were not as robust as they possibly could have been and that that had affected the veracity of the overall results claim.
Emma Back: That is right. We understand why DFID and many other agencies working in this area use modelling for these purposes. The data locally are not as available or robust as they need to be, and that may change over time with the investments that are being made in local health management information systems.
Our concern was about whether the estimates had been properly interrogated, as much as anything else. There were a number of other sources of data that, when triangulated with the estimates that had been produced through DFID’s modelling, might have suggested that they were heading towards overestimation. They were also significantly out of kilter with the original forecasts that were put together for the bilateral aid review at the start of the results framework development process. We think orders of magnitude of 387 being the original forecast for Malawi and 10,100 being the ultimate estimate might have raised some alarm bells. There are various other modelling tools used by some of DFID’s implementing partners, including in Malawi, which were generating figures of around half those generated through the tool that DFID had selected. Again, there were alarm bells being rung that might have been followed up. Our recommendation in this area is that DFID could use other quantitative and qualitative data to assess whether the assumptions of its modelling are appropriate and whether the resulting estimates are robust.
Q7 Chair: In DRC in particular, you estimated that you have saved 4,800 maternal lives in that four-year period. That is based on women delivering in a health facility. Some 48% have access to comprehensive emergency care with a further 12% having access to basic emergency care. Do you think that is overly optimistic in the DRC context?
Richard Clarke: Would you mind if I quickly came back on the triangulation point?
Chair: Please do, yes.
Richard Clarke: I might ask Will to say a bit about DRC. There is an important point to make at some point here, so I will make it now. In each of these countries, as Emma says, specific aspects of the way in which we have done our modelling will vary. There will be specific elements of the way we have done modelling in individual countries, because we and our partners have used different techniques. I have no doubt whatsoever that there will be specific issues and challenges about almost any country in which we work.
We are huge fans of triangulation in DFID, as an approach to ensuring we can back up the important and robust claims we make about the impact of taxpayers’ money spent through UK aid. In the case of these programmes, it included things like comparing with other estimates that had been done locally and following up with field surveys, for example, whenever we can. There was a particular modelling tool that we used, which has been internationally recognised and endorsed. In Bangladesh, for example, when we carried out further analysis based on that form of triangulation, we were able to use that to finesse the modelling. A lot of that work has already gone on, but we recognise and fully accept the recommendation that there are always new forms of triangulation that could be used. There are new forms of digital work, for example, which we can use to make this better, so we will always look at opportunities, whenever we can, to triangulate these kinds of data. The recommendations we have had from ICAI on this are helpful.
In relation to DRC, this particularly swings on a question about how we defined what a reasonable form of care would be in particular settings, but I will ask Will to say a bit about this specifically.
Will Niblett: We completely accept the recommendation that we ultimately need to be investing in data at country level. We ultimately need good quality data to put into these models.
On DRC, as I understand it—and come in if this is not the point—the assumption was that, because only one facility met the basic emergency obstetric care standard, it did not appear credible that we were claiming the results we were. The point here is that the basic emergency obstetric care standard, although it has the word “basic” in it, is still a very strict standard. In a country like DRC, you may have a whole load of facilities that nearly meet, but do not quite meet, that strict standard. That is what we maintain happened here. Just because you are not meeting that standard, it does not mean that you are not saving lives in that facility, and that fed into our model.
Dr Evans: The programme impacts were modelled on a set of assumptions that involved this service being available, so I do not see how that works in this case. We acknowledge that, post the 2015 period in DRC, there has been an improvement and expansion of these basic facilities, which is a good direction of travel. At some point, it will be important for DFID to go back and find out exactly what is being delivered on the ground, in terms of both basic obstetric and emergency care, and maternal deaths averted. At the time period we were looking at, the figures they were coming up with simply did not look plausible for what was actually happening and being delivered.
Q8 Chris Law: A lot of the points have been raised and answered, but the most important thought is public trust. When you are using modelling, is it “one size fits all” or country specific? Secondly, you talked about the need to review it, make sure we are up to date and triangulate. Do you have a programme to do that every six months or every year to make sure it is continually being tested?
Richard Clarke: There are two important points here. I will take them in turn. First, you are right that, for us, the trust in the methodology we use and the outcomes we deliver is fundamentally important. The reason we have put, for a number of years and in different ways, such an important focus on being able to communicate our results is precisely that. We are custodians of a very large sum of public money, which the public expects us to spend in a way that is appropriate and efficacious, so it is an important part of what we do.
There are a number of things we are trying to do for that reason, and this has been a rolling programme. As Matthew Rycroft said at the last ICAI appearance he made recently, this is something we are looking at, at the moment. During 2019, we are going to say more about the approach we are taking to our results in general and collectively, but I will raise a couple of points quickly now.
We are already completely transparent about the methodology that we use. We do that in a completely open way, in part to prompt challenge, not just from ICAI but from academics, practitioners and others in the field, who think there are ways in which we can improve the approach that we use. We also now have a one-stop shop access point on our website for anybody who wants to look at our results, the public, journalists, academics and others.
The final point to make on this is that it is important, with respect, that we do not lump together potential distrust in what we do with modelling. We would want and I would want, today, to make a robust defence. I am pleased to hear Emma say that there is nothing wrong with modelling in principle. I think ICAI, while it has concerns in a number of cases about specific aspects of the way in which we have done modelling, would acknowledge the principle that modelling is the right thing to do to demonstrate the outcomes we have had. In circumstances where, for a variety of reasons, such as the country not having collected the data we would ideally want, we cannot use the robust techniques we ideally want to, I hope we could collectively push the importance of modelling as an appropriate technique. That is not the same as saying we would use the same approach everywhere; we use different approaches in different countries and settings. It depends on a number of aspects in the minutiae of how we are doing that. Of course, the risk in that is that we potentially identify alternative ways in which those individual elements of modelling could be improved.
You asked specifically about the frequency with which we do that. We have a rolling programme, rather than a specific six-monthly approach. There are two particular ways in which we update our figures. The first is if we come across something that suggests a potential change, such as an alternative form of triangulation. The second, which is much more common, is when a new form of data comes on stream. We are working with a number of countries at the moment to help them develop better data systems locally. If that then allows us to feed more in, we always do so. Country offices are well aware of that approach.
Q9 Chris Law: I have a last question on this. Two countries have been picked out. Have you found other countries, from your own experience, where your modelling has been incorrect, and incorrect on the side of overoptimism or unrealistic objectives?
Richard Clarke: I would respectfully, to be clear, dispute the suggestion that the modelling was incorrect. We have certainly found situations in which we have changed the ultimate answer that our modelling was suggesting, because of new inputs. A model will always be based on a set of assumptions and, where those assumptions change or we get better information, it might change. It was incorrect for us not to have reported the change once we had established it, as I said at the start, but there have been situations in a number of other countries where we have used assumptions that we have been able to improve on. In some cases, those have shown an increase in the efficacy of projects and, in some other cases, they have shown a reduction.
Overall, wherever we can, we make sure we are using robust but conservative assumptions. We are trying to make sure that we are not using radical assumptions. Secondly, we try to make sure we have a blend, so that overall we can step back and say our headline numbers look appropriate, because they are made up of different kinds of modelling, but at appropriate levels.
Will Niblett: One example of where we have taken a deliberately conservative approach to this is that our figures do not include our multilateral contribution. We were keen not to double count, so throughout this process, wherever there has been a suggestion we might be counting something twice, we have discounted that, so that is another example.
Q10 Chair: I will come back to you in a minute, Alison. I will ask one more question of you, Richard. ICAI uncovered various problems at DFID‑supported health centres, including poor record-keeping, all clients being recorded as new, clients being double counted and given a new record if their name has been mis-spelt or if they have moved to another provider. What are you doing to resolve those problems?
Will Niblett: First, that was an upsetting recommendation for us to hear. As a global community, we need to do better in how we are supporting the gathering of data in facilities. There are challenges in health facilities across the world, as the Committee will be aware. In Malawi, we are now funding an electronic system. That is one example of the various investments we are making to improve record-keeping. It is worth saying, for all the challenges in Malawi, it is a country that has had quite a lot success since 2010 in the availability of contraception. This is a family planning comment, I believe. They are still making great improvements in spite of those challenges.
Chair: Is there anything you want to come back on, Alison or Emma?
Dr Evans: We need to be clear that under all this discussion of modelling and so on are people’s lives. We need to make sure we focus on them, and I know that is what concerns DFID most. There were just a couple of things. Modelling as an enterprise is legitimate, but it is only as good as the assumptions and the quality of the data going in it. Our continued concern is that that was not looked as thoroughly and rigorously as it should have been. We have two things on that. First, a lot of data was front loaded, so lives saved in the future were incorporated into a claim for the 2011-to-2015 period. That is one issue.
The second, which was also identified in the model’s own final report at the end of 2015, was that some of the assumptions that hold at a global level may grossly overestimate impacts at a country level, because they do not hold there. I take Richard’s point that some of those were looked at in context, Kenya being one, but we did not see that in Malawi or DRC.
Q11 Stephen Twigg: My questions follow up on what Alison has just said and are about data collection. ICAI, in its report on Malawi, referred to data use not being “mainstreamed across district”, and expressed concern about the lack of the use of data at a national level to inform policy guidance, planning and so forth. Can I ask DFID? DFID Malawi is supporting district health teams to collect data about maternal health, but as I have just said, ICAI found no use was made of the data, so what was the incentive for the district health teams to continue with collection and what is being done now to address this?
Will Niblett: Your question was specifically about Malawi, I believe. I might need to come back on the specific progress on this in Malawi.
Richard Clarke: The first thing to say is that we are working in a range of countries to do two things. The first is to improve data collection nationally and regionally. We do a lot of that directly through work with Ministries of Health and equivalents. You asked specifically about incentives. The short answer is that, on the ground, we are often working, as you will be aware, through a chain of delivery, which in some cases is a mix of private and public providers, but in many cases is based on federated substructures locally. We will never be able to establish a perfect system locally, particularly when we are working in situations in which we are a core funder but alongside a public sector provider that is the national provider. There are two or three points in the ICAI report where, entirely appropriately and perfectly rightly, ICAI challenges us on the precise aspect of the way in which a particular country, Malawi or DRC, is delivering its healthcare locally. Part of our challenge is that it is up to the Government of Malawi or DRC as to how they do it.
The honest answer to your question about incentives is that DFID Malawi has been working with the healthcare providers locally to make sure we are emphasising the importance of good data collection and particularly data disaggregation. When I last appeared before the Committee, I was talking about disability and emphasised the importance of data disaggregation as something we are really pushing. First, we are emphasising the direct utility of this; it is worth collecting it because you can help us save lives. Secondly, it is through good training.
Q12 Stephen Twigg: Is there evidence in Malawi that things are changing as a result?
Richard Clarke: As a result of the data that the Government of Malawi have been collecting since 2011 to 2015, as Will said, yes, there is a lot of evidence to suggest that healthcare outcomes in Malawi are improving.
Stephen Twigg: You look sceptical, Alison.
Dr Evans: I feel a little sceptical about that. We would all wish that, absolutely, but the UK has been involved in Malawi a very long time. It has been, if not the top, one of the top bilateral donors for a long time. There are many challenges—governance and public financial management challenges—to working in Malawi. The DFID team is working under quite challenging circumstances, but it is interesting, on local data collection, that we saw Malawi as one of the bright examples of taking data collection right down to facility level and communities in this programme called Evidence into Action. We saw the footprint of Evidence into Action gradually disappearing, because it is a programme that is no longer supported by DFID. It was picked up for a year or so by the Gates Foundation, but is no longer supported. It was filling these gaps in local data collection, providing really useful information, both for communities and for providers. It is disappearing before our eyes and has no future funding. It seems such a tragedy because that was, in many ways, an excellent response to a really challenging situation in a country like Malawi.
Q13 Stephen Twigg: Can I hear a response to that? It sounds pretty damning.
Will Niblett: As we have said before, Malawi is one of the poorest countries in the world. Some 70% of the population is below the poverty line and there are huge challenges. In recent years, we have been trying to focus on quality of care as well. We will come on to that, but the use of a mystery shopper approach to assess quality of care is well embedded there now.
Q14 Stephen Twigg: Tell us about the specific programme that Alison said was working and is now no longer being supported.
Will Niblett: Our support for the Evidence into Action programme has indeed come to an end, and we are now evaluating what the next step should be and where to take that forward. I know that other partners are continuing to support it, but not in Malawi, so that is something for us to look at, at the next stage.
Q15 Stephen Twigg: If the evaluation shows it was a great programme, is there the possibility of going back and starting to support it again? Does that not raise questions about why funding was withdrawn from a programme that, as Alison has said, was very good?
Will Niblett: Rather than “withdrawal”, I would say it has come to the end of the programme, so now we have a point—
Stephen Twigg: If we are no longer funding it, we have withdrawn from it. It is just a different way of saying the same thing.
Dr Evans: You stopped funding it in 2016.
Will Niblett: In the new phase, we will look at what the next stage should be for our work in that area.
Richard Clarke: At the global Family Planning Summit 2017 we, along with 60 other international donors, signed up to a global programme to improve data collection in a range of countries across Africa. We think that is a good approach to solving some of these problems, but I accept that there will always be specific instances where we were doing particular things that may have drawn to a close.
Q16 Chris Law: Moving away from methodology to statistics for a minute and looking at your delivery, in DRC only one DFID-supported facility was able to provide basic emergency care in 2016. That is one against a target of 146 and, in 2017, you were still quite a way off track. What is currently being done to resolve this situation and how quickly will you resolve it?
Richard Clarke: The first thing is to come back to this point about the single facility, because we talked about this a bit earlier in the questions. There was one facility in the period that the ICAI report covered in 2016 that met the strict technical standards for providing emergency obstetric care, but that does not mean other facilities around the country were not delivering elements of quality care. DFID support has seen a significant increase in that, by 2017, 143 facilities met that standard. That was partly because we put a strong focus on training providers, so a lot of this is about training. Our approach to quality is one of the important aspects of the way in which we do this work. We are focused on direct delivery of elements of programmes, making sure that facilities are up to scratch and ensuring that the individuals who are working in them are properly supported. That training, which was principally done during 2016, so just after that period, has been effective. There remain big challenges to working in DRC for a variety of reasons, but we remain committed to supporting that improvement and we are confident it will continue to deliver more.
Q17 Chris Law: Correct me if I am wrong, but the target in 2017 was 177, so you have seen 143 against 177. What is the target for 2018 and how close are we to getting there?
Will Niblett: It might help to talk a bit more about how we moved from one to 143. As I said earlier, this is a strict standard. After we got to the point that only one facility was meeting it, we looked again at the appropriateness of the way we were implementing those standards. One of the issues is that you have to see a certain number of people to qualify. In a country like DRC, the facilities were not getting the number of clients, so we revised that approach and took that away, which helped those other facilities to meet that standard and gave us a better reflection of the level of care that was going on in DRC. Now, it is obviously a challenging environment to work in, as I do not need to tell you, so we now have a new phase of programming where we are also looking at quality of care in DRC, including working with the Ministry of Public Health in training staff to deal with these data reporting issues.
Q18 Chris Law: My question was what the target is for 2018 and how close we are to it.
Richard Clarke: I am sorry; we might need to come back to you on that, Chairman.
Q19 Chris Law: Again for DFID, in Malawi only 15% of DFID-supported facilities were able to provide basic emergency care in 2016-17. I have a similar question: what is being done to resolve this situation? Again, it is quite a woeful figure, so has it improved significantly since that period?
Will Niblett: These are still challenging environments to work in.
Richard Clarke: The short answer is yes. We talked about this a moment ago. Both Malawi and DRC, and most of the countries in which we operate these kinds of programmes, are challenging but, as we said earlier, Malawi has seen significant improvements in healthcare and the outcomes we are talking about here. The levels of contraception have been delivered and the rates of maternal survival have improved so, yes, we are confident we are making progress there. We do not have a specific target for this now. We have changed the approach that we take since 2011 to 2015.
Chris Law: You do not have a target.
Richard Clarke: We do not have a specific target in Malawi. In the 2011-to-2015 period, we had the particular targets that we are talking about today. We are now approaching this through the lens of family planning and we are making real progress there.
Q20 Chris Law: I hear all the right words—there is real progress being made—but if you have no targets, what do you measure against? Why are there no targets, given that you have used targets previously?
Richard Clarke: We have an approach to results in this area based on ensuring that we are delivering, originally based on a set of approaches that was focused specifically on maternal health and newborn lives saved. We are still focusing on those areas now, but in relation to family planning more generally. We regularly publish a set of results, but the targets have shifted during that period.
Q21 Chris Law: Have they shifted or been removed? I am trying to be clear here. You said there were no targets. Are there targets or not?
Richard Clarke: I am sorry; I meant in that specific area.
Will Niblett: It may help to distinguish between different targets. We no longer have global-level targets. Malawi and DRC have their own specific programme-level targets, which will be used to monitor the progress of that work. I apologise; I do not have that precise information to hand. For both Malawi and DRC, we can come back to you and give you a specific answer to those questions for those countries.
Q22 Chris Law: During ICAI’s visit to Malawi—this is quite shocking to me—it is clear maternity wards were under considerable strain, where you had large numbers of pregnant women queuing outside for a bed. I am sure that shocks all of us in this room. What is being done about that in Malawi and what is going to ensure that there will no longer be a single queue or a person waiting in a queue?
Richard Clarke: We were also disappointed to read that result. Will said earlier we found it shocking, and I know the team in Malawi did as well. One example that we have specifically used in Malawi, in our family planning programme, is the mystery shopper approach. We have used that to assess both the quality of care provided and the behaviour of health workers locally, because one of the themes that came out in the ICAI report was concern about the behaviour of healthcare workers. We have been able to use that as a quality assurance indicator to ensure minimum standards are met and, where they are not, that prompt action is taken. That is one example of how we have sought to use that approach in Malawi.
Q23 Chris Law: I have my own thoughts on this, but I am going to hand this over to ICAI for responses on both DRC and Malawi. What are your reflections on the answers?
Dr Evans: I will say a few things about Malawi, because I visited as well, so it is fresh in my mind, and I will ask Emma to say a bit about DRC. The responses from DFID point to one of the challenges that, in Malawi, the vast bulk of the programming that supports the maternal lives saved claim is family planning. It is prevention of unwanted pregnancies. How much they are doing through facilities is extremely small, despite the fact that we know 70% or more of maternal deaths are to do with complications during pregnancy and childbirth. They cannot answer some of your questions, because they are not doing that direct facility-level support in much of Malawi. They had a programme that was looking to support that, but they largely had to bring it to an end because of the corruption scandal in Malawi. That meant they could no longer put funds through the Government Budget. Ever since, they have been trying to provide workarounds for that, in a sense. That means they do not have a line of sight to the problems we have identified, which Emma and I think they should have, if they are actively working on maternal health issues.
The point about 15% of facilities is in a DFID annual review and relates to the Health System Strengthening and Support Project that it had in place, which can no longer distribute funds directly into the Government Budget. Importantly, when we were in the field, a number of facilities had had this accreditation to provide basic obstetric care, but could no longer claim to have it. They were in regression, so the picture was not improving at all. If anything, it was continuing to get worse and that raised a number of major alarm bells for us, particularly when you weigh it up against the large number of lives saved that is being attributed to this. We are not questioning the real value of family planning, but it did not fit together as a comprehensive picture for us.
Emma Back: Shall I carry on in relation to Malawi? One of the reasons that the pressure on facilities exists is that the Government have set high targets for achieving facility-based delivery for women, so they are encouraging women to come to facilities at a higher level of the system to give birth. Malawi has developed a community health strategy recently, which is designed to take some antenatal and maternal health services down to community level. That is an area that could potentially be further supported and might relieve that pressure.
The health sector support programme that Alison referred to has a service-level agreement, through something called the Health Services Joint Fund, with the Christian Health Association of Malawi. These are faith-based facilities. That is designed to ensure that women are able to access antenatal and delivery services for free. Again, it addresses the access issue, not the quality of care issue. There is something here about ensuring that, as you expand access and encourage people to go to facilities, you address the quality of care they receive there and ensure those facilities are adequately staffed and equipped.
Moving on to DRC, we saw, again, some perverse outcomes from Government policy there. There were lots of junior nurses in facilities, but very few midwives and doctors, so there is a human resource imbalance there, and facilities are poorly equipped. DFID, through its Access to Health Care programme in DRC, has a programme of refurbishing or constructing new facilities where refurbishment is not possible. At the moment, it is lagging a long way behind its targets, but we saw some promising work being done. For example, the lead implementing partner for that programme in DRC has been designing new health facilities and taking on board women’s views to ensure that those new health centres are better designed to accommodate women who go in for antenatal maternal health. There are some pockets of good work going on; it is just a question of whether it adds up to transformative change and a real improvement in outcomes for women and their babies.
Richard Clarke: This is a really difficult area. As we scale up our support and as more facilities open, we are rightly holding ourselves to an internationally high standard for the quality of care that is delivered in those facilities, and ICAI is challenging us on that. I have a couple of points on that. First, we were recently part of a global commission on quality care standards that The Lancet led in October 2018. That has given us some of the best evidence of what works, in terms of scaled-up quality of care. We have supported and signed up to a declaration as part of that, and we are going to be providing more guidance as a result of the findings that come from that, as part of all our health programmes going forward.
We are already mainstreaming quality of care into our bilateral programmes around the world, more generally. From a sample of 45 bilateral programmes in place this year, 28 explicitly stated that they were addressing quality of care and that was before we stepped up our country-level and global contribution work. For us this is a really important area, and Alison and Emma have each identified good examples of things we are doing in both Malawi and DRC to try to get to the heart of this complex problem.
Will Niblett: I have some examples. In Malawi, we are working with traditional leaders on the social barriers to women accessing care. Some of the things I found most upsetting in the report were around disrespect, discrimination and abuse in facilities, so we are looking at how you work with people to address that through the workers in the facilities, because there is a culture in the facilities, but we are also working on issues of how women are perceived that may stop people from accessing services.
Chair: We need to keep the pace going, Henry.
Q24 Henry Smith: I will try to keep the pace going. Thank you, Mr Chairman. Continuing on the same area with a question to DFID, the UK provides significant resources to the UNFPA. However, in 2015, only seven of 46 countries in which the UNFPA works were able to confirm a regular supply of contraceptives in at least 60% of their facilities, while none of the 46 countries was able to confirm the availability of key maternal health medicines, such as magnesium sulphate. What, in DFID’s opinion, is going wrong?
Richard Clarke: The supply of medical supplies is an enormous challenge globally, and we are at the forefront of work on supplies and supply chains, but we recognise that getting this right is a global challenge. We are supporting contraceptive supplies in a number of different ways. As you mentioned, it has been a challenge for the UNFPA, but we continue to work with the UNFPA to procure contraceptives effectively. One of the ways we are doing that, as we do with GAVI, the Global Alliance for Vaccines and Immunisation, is making sure that we can support volume guarantees. We work to ensure that countries can buy, safe in the knowledge that those volumes are going to be there, making sure that we are reducing price as a result, on a permanent basis.
Will might want to come in on this, but we talked about data earlier. We are also involved in the large-scale sharing of data on contraceptives, between manufacturers, procurers and countries, on a single IT platform called the Global Family Planning Visibility and Analytics Network, which allows sharing of that kind of data. We hope, although this is a challenging area, that things like that will make the direct supply of contraceptives and other forms of medicine to the individual who needs them more effective.
Will Niblett: You asked what is going wrong. The global community is quite good at getting it to the country, but getting it from the port into the hands of the women who need it is the challenge. With things like the global VAN that Richard just mentioned, the long-term aim is to have real-time checking of what is in the facility and at all points along the supply chain. We are investing in things like that, as well as bringing down the price. Then there is the social work that we talked about earlier. Many of the barriers to this are about acceptability for the women. A woman may think that, if she goes to the clinic, she is not going to be treated respectfully, or she may have heard scare stories about some of the products. It is a deep-seated issue that we are doing our best to address.
Q25 Henry Smith: I have a few quick supplementaries again to DFID. The central warehouse in Malawi that DFID is supporting had only two of the 12 key maternal health commodities in stock, in 2015-16. How is that justified?
Will Niblett: It is not. It is unacceptable and that is why we are making these investments to change the situation.
Q26 Henry Smith: In DRC, essential supplies such as condoms and pills are often unavailable at the local facilities supported by DFID. Again, do you disavow that as acceptable?
Richard Clarke: Yes, absolutely. I am confident in saying that, in almost any of the countries where we were working during this period, there will have been significant issues around supply. That is precisely why we are doing this work, including through the Family Planning Summit in 2017, where collectively and globally countries identified significant blockages, particularly in countries in getting from the port. That is why we secured extra commitments from those countries to work on those supply chains. Yes, this was absolutely a problem during the period that ICAI addressed.
Q27 Henry Smith: I appreciate that we are talking about 2015-16, two or three years ago. Are there any signs of that situation improving?
Richard Clarke: Yes, both through the commitments that we are making in the way I have just described, and by working more closely with our partners in individual countries to make sure those supply chains are addressed.
Q28 Henry Smith: Finally, I appreciate you are sitting next to each other and am sure there is no awkwardness but, to ICAI, are you satisfied with DFID’s responses in this area?
Dr Evans: I have a couple of things, and then Emma may put in one thing. You started with the point about UNFPA. DFID is a very significant supporter of UNFPA and it has an important role within the international architecture. Our feeling is that, with UNFPA’s presence at country level, even more could be done to leverage its influence over the issues that we have just been talking about but, perhaps more than anything, around the key target groups for this support in the family planning area. We visited the central medical stores in Lilongwe when we went out there and I think things are on an improving path. However, when we visited a number of facilities, there were still stock-outs of some key drugs, such as oxytocin, and women said to us they were not given a choice of contraception. These are still issues, but we will park that as anecdote for now. These stories need to be reconciled at some level; we need to see these things come together.
Emma Back: I echo those points around the need to secure DFID’s huge investment in the UNFPA by ensuring there is good working at country level between UNFPA and DFID, and other partners. This is largely a distribution challenge. Again, it comes down to facilities, to forecast their needs, to make orders at the right time and to understand the needs of the communities they serve, and then work their way back up the supply chain and the security of that supply chain, ensuring that products do not go missing, for example. It is a huge challenge, but it urgently needs to be addressed because, at the end of the day, if a product is not there, a woman, a girl or, for that matter, a man or a boy does not have a choice, and that is what we were hearing time and time again.
Q29 Henry Smith: You are saying that the key problem is the supply chain. What is happening to the supplies when they arrive at the port and do not get fully to the women who rely on them? What is happening to those medicines and products?
Richard Clarke: It varies—that is the honest answer—depending on the countries in question. In some countries, there are challenges around just getting the stuff out of the port or delivery place safely. In other countries it could be to do with conflict. There are probably two key things we are trying to do. We have talked a lot about what we are doing locally, but there are two key things we are trying to do globally and at country level. At the global level, ICAI is right to say and we fully accept the recommendation that we can be doing more to work with UNFPA, for which we are a significant funder. There is more that we can do to leverage its position globally and in countries. I accept that.
At the country level, we are trying to do two things in parallel in every country where we work on health programmes. The first is to get right down to the individual people who need care, alongside local healthcare providers. The second is to be working in the capital city alongside the Health Ministry to say, for example, that we can use data, geographic mapping, poverty mapping analysis and things like that to work alongside the Health Ministry to make sure that it understands what is happening to get these flows going through. That is fundamental to our approach more generally.
Henry Smith: I could question further, but I know time is pressing.
Q30 Chair: Thank you, Henry. In Malawi, there was a programme between 2009 and 2015 that enabled 271,601 additional women to use modern family planning methods in that six-year period, but the target was 466,187. I am just wondering why it failed so badly and was so short of the target.
Richard Clarke: I will come in first; then Will might want to. This was a really complex and difficult programme. Reaching women in a way that allows them to make informed choices was extremely difficult. We continue to work in this area and we are working with community-based outreach services. We are trying to see what we can do to reach women in poor and rural areas, but this was a difficult programme and, ultimately, we were not able to deliver the results that we had hoped in that area. Will might want to say a bit more.
Will Niblett: It is just that Malawi did in that period manage to raise its contraceptive prevalence among married women to 60% from 42%. Even though, sadly, we missed the target, there was still progress.
Q31 Richard Burden: Staying with family planning in Malawi, you said it is an area where some progress had been made. I am conscious that initial family planning services are provided for free, but around 30% of women then have to pay to have either intrauterine devices or contraceptive implants removed. DFID supported a voucher scheme to help with that, but it came to an end in 2015, so these are my questions. Did it come to an end because it was not meeting its objective? If it was not meeting its objective, what was the objective? What other ways do you think that objective could be met? If you thought that programme was reaching its objective, could it be reinstated?
Richard Clarke: Will might want to comment on this, but I think I am right in saying that we did not end the programme because it had not reached its objective. We finished it because it had reached the end of its period as a programme. You asked specifically about the case of women being charged for the removal of contraceptive devices. This is a difficult area, and the NGO that we work with has a policy of subsidising its work with the poorest alongside the private clinics that they run. The community-based outreach services that the NGO we work with operates, which often serve the most rural and the poorest, provide both insertion and removal of contraceptives for free, but there are options available for women who pay to have them removed. That is always going to be a balance, which we think local NGOs are well placed to determine, but it is a sensitive issue, and we work closely with them and would always want to keep it under review.
I said earlier that we are broadening our approach in Malawi to family planning in the round, as part of the approach we are taking, building on our long history of working there. We are seeking to deliver the same kinds of outcomes, but in a broader context.
Q32 Richard Burden: Is the proportion of women who need to pay to have implants or intrauterine devices removed coming down or staying the same?
Richard Clarke: I do not know; that is the honest answer. I do not know what the latest stats are, I am afraid.
Dr Evans: Across the piece, we understand that DFID has stated in many ways a commitment to try to minimise or remove user fees for a whole range of basic health services, of which maternal health services and family planning are one area, but we found a lot of inconsistency. It is difficult to monitor and keep a track of, particularly when you have a wide variety of implementing partners with possibly quite different business models for how they are delivering. We would like to see, if I put this positively, a much more consistent stance built into DFID’s position on its support for the health sector and universal health coverage in general, rather than a patchwork approach based on the different business models of different providers. That is what we did not see clearly articulated, and we are waiting for this health system strengthening position paper, which is mentioned in the management response. It has been a very long time coming. This could be an area where DFID commits to building a consistent approach to its position on user fees, particularly for key target groups.
Will Niblett: We have a varied approach to user fees more generally, because we deal with the situation we find on the ground. Our policy is to take what we have on the ground and make it more pro-poor. We are encouraging the removal of user fees, but also working in the context we are in. We agree that the long-term objective here is a strong health system in country that is funded through taxation and other means, so that the burden of expenditure does not fall on poor people as it does at the moment. That is what we are working towards.
Q33 Chris Law: ICAI clearly argues that younger mothers are at a greater risk of complications and may be more likely to suffer discrimination and abuse, yet we find that only three of the eight centrally managed programmes examined by ICAI explicitly prioritised both poor and young, or otherwise marginalised, women and girls in their original programme documentation. ICAI has also stressed the importance of involving younger adolescents aged 10 to 14, yet your key target area seems to be 15 to 19. What do you say to those positions?
Richard Clarke: We acknowledged that this was a weakness. We have said that in our management response. We recognised that it was a weakness, but it is an area on which we have now increased our focus. There are two forms of marginalised groups that we are keen to focus on in this area and that ICAI has rightly challenged us on. The first is making sure we are focused on the very poorest in particular countries and the second is around adolescents. I will just say something about what we are trying to do around poverty; then I might ask Will to say a bit about the precise nature of the work we are doing around adolescents.
We are using national poverty mapping data, wherever it exists, to ensure that we are targeting the poorest areas. That is challenging. It depends on the form of data collected nationally, which is why the work we are doing in particular countries around data is important. In Pakistan, for example, our provincial health and nutrition programme is increasingly designed to work in the poorest areas of Pakistan. In Zimbabwe, we have used the poverty atlas report to ensure that we are deploying national volunteers and village health workers to identify the poorest women and their health-related needs in key areas. That is as a direct result of the uplift we have sought to do over the last couple of years. Do you want to say a bit about adolescents?
Will Niblett: As Richard says, we acknowledge that this has been a challenge for the global community for a long time. At the Family Planning Summit we held last year, not only did we commit to disaggregating all our data, but we got 60 other partners to do the same, so we are leading the world on that. It is still a challenge to embed it in country systems, which is obviously where we need to get to. We would love to do that for 10 to 15.
Q34 Chris Law: Have you changed your key target then? That was the specific question.
Will Niblett: That target was from the previous period. The reason we have initially focused on the older adolescent group is that it is a huge challenge to get that data embedded in country systems. That is the first step. If we can get that embedded, the younger adolescents will be the next challenge.
Q35 Chris Law: Speaking about adolescents in particular, there has been little reference to adolescent boys in any of your centrally managed or country-level programme documentation that has been assessed. Why is that?
Will Niblett: We have focused particularly on adolescent girls, because of the focus on maternal deaths and the consequences for them. We focus a lot on adolescent boys.
Q36 Chris Law: Should the focus not be on both?
Will Niblett: Yes, absolutely. We focus on adolescent boys particularly around the gender norms that are being developed, but we take on board that, going forward, we need to look at both.
Q37 Chris Law: Lastly, before I come to ICAI’s responses, “In Malawi, the DFID programmes we reviewed did not appear to have engaged directly on policy or legal issues related to harmful practices, such as forced sexual initiation, which remains prevalent in Malawi and is associated with unintended pregnancy and sexually transmitted infections”. Why not and is there even an education programme underway in local communities? What has been done?
Will Niblett: We do engage on sensitive issues, but it is sometimes harder to have visibility on that, because they are potentially sensitive issues with the Government locally. If you look at our work in Nepal, for example, we have worked for a long time on reducing dangerous backstreet abortions and we have seen some policy results there. We work on comprehensive sexuality education in various countries. I spoke before about our work in Malawi with religious leaders on norms and harmful views about the way women are perceived. We have just announced a large FGM programme, building on what we have been doing for the last few years, which will all be about tackling with communities and supporting African leadership to change these difficult and sensitive issues in country.
Dr Evans: It was a clearly stated commitment from DFID at the beginning of this results framework period that it was looking to achieve a particular set of results for the poorest 40% of women and adolescents 15 to 19. At the end of that period, they could not report on either. We just need to be clear. I do not doubt and we would not doubt that there is now a lot of commitment to try to put that right but, at the time, that was not reported on and they simply could not do it in the form that had originally been hoped for.
Since then in our engagement with DFID, particularly post the 2017 Family Planning Summit, we have seen a real galvanisation and commitment to target adolescent girls particularly. That is positive, but it is still at the level of “we will” as opposed to “we have”, and we need to keep that in view. The initiatives that Will has just mentioned are all valuable, but what is pulling all this together into a clear strategic thrust on reducing maternal mortality and morbidity? Where is that bit? Yes, social norms are important. Yes, FGM is massively important, but where is that coming together? That is what we are missing, in a way, because it is not clear at country level and that is a missing piece.
Will Niblett: I can come back on that.
Chair: I will give you a chance at the end.
Q38 Henry Smith: This again is to DFID. ICAI argues that, by your reluctance to channel funds through Government-funded family planning programmes, you are prioritising short-term results over long-term sustainability and potentially disincentivising Government-run health facilities. What would you say in response to that?
Richard Clarke: I feel like I have said this a fair bit today, but our approach is to make sure we are taking a balance in the context of the individual country. We provide support to a range of providers, both state and non-state. It depends on the precise nature of what we are trying to achieve. We always want to work through a state provider whenever we can, particularly in circumstances in which we are managing the downside risk of crowding out. All our support, though, is aligned to national health sector plans, so we are keen to make sure we are doing that appropriately, but in these particular cases we do not believe we have created a situation in which we are incentivising the state out of local provision. That is something we try to bake into our approach.
Will Niblett: There is not a binary choice in many situations. We would be working with a non-state provider to support the capacity of the state. A good example is in our support for a global financing facility in northern Nigeria. We are working with private providers to incentivise them to work more effectively with the local authority there and to help the local authority hold them accountable for their service delivery. It is more of a dynamic relationship than it sometimes first appears.
Richard Clarke: If you take Malawi as an example, it has had a long and mature history of a relationship between the state and both state and non-state providers locally. We see working with non-state providers in Malawi as an enhancement, rather than displacement, of the state.
Q39 Henry Smith: Quickly, because I am conscious of time, this is on Malawi and to DFID. DFID contributes to a Health Services Joint Fund, a pooled funding mechanism, outside Government financial systems. However, ICAI discovered that this fund is being used to cover things like utility bills, rather than to invest in the development of health infrastructure. Again, with apologies for being trite, how can that be justified?
Richard Clarke: I am not familiar with the precise details about that and might have to come back to you, Mr Smith, if that is all right.
Q40 Stephen Twigg: ICAI states in the review that there has been a loss of institutional memory with frequent turnover in DFID staff, implementing partners and external stakeholders that work on maternal health. Do you accept that and what are you doing about it?
Richard Clarke: I have been in this job for four and a half months, so I do not know if I am the right person to talk about turnover of staff.
Q41 Stephen Twigg: That demonstrates the issue, does it not? How long was your predecessor there?
Richard Clarke: It was quite a long time. Will and colleagues who are here today have been working on this for a long time. This is a common challenge in the public sector, bluntly, the turnover of staff. We are committed to DFID being a learning organisation. My sense is that one of the most common thematic concerns ICAI has is whether we are really learning collectively. It is something that Matthew and the board are conscious we need to make sure we are doing well. I will make a couple of observations.
First, through the country office relationship with centrally managed programmes, such as those you have heard about today, we are committed to making sure that the experience of what has worked previously and how these programmes have been run is properly shared as part of induction and things like that. The second thing is making sure, putting it in a basic way, that we do not just reinvent the wheel. The biggest risks within institutional memory turnover are, first, that we end up missing things that have previously worked, and secondly that we just start everything afresh every time. It is an ongoing project, but I do not think that we are suffering from that problem. It is not impacting in a negative way.
Dr Evans: My sense is that there are challenges here for the Department. I do not think that you do not know about them; we have talked about them, in this forum, on a number of issues. Particularly in this arena, we were conscious that a sense of accumulating learning and insight, and building on that, was not very evident. We were concerned that some things that were considered good practice and key areas for DFID in the past were clearly no longer part of the programming or the way forward, and we were not entirely sure how that came about. Staffing is under pressure in DFID, as it is across the public sector, so it is clearly a demanding environment in which to do this kind of learning and building on experience, but we see no reason why we should not keep up that pressure.
Richard Clarke: Do you mind if I just come back on that? There are a couple of specific things we are doing at the moment. It is probably less likely that, in a particular area of programming, people will fail to understand what has happened previously, although there is definitely a risk of that. One thing for which we do not have as strong a story to tell as we would like is making sure we are learning lessons from different kinds of programmes. In my view, we have been rightly praised by ICAI and the IDC for the work we have done on violence against women and girls. One thing we have sought to do, not least through events that we have organised in DFID for colleagues—lunchtime seminars, lunch and learn, things like that—is to make sure that the key lessons about why the approach to violence against women and girls worked well are shared collectively.
Secondly, there are people dotted around the organisation who are charged with making precisely these kinds of connections. I oversee the heads of profession, so the professional career managers for all the governance advisers, healthcare advisers or education advisers around the world, in different DFID country offices. When I met them recently, we talked about how we could make sure we were communicating best practice as rapidly as possible, so we are doing things to try to address this, but it is undoubtedly an ongoing challenge.
Chair: Thank you so much, all. It was a good, robust conversation. Again, Alison, good luck to you in the future. It has been a pleasure to work with you for the last few years.