Submission to the International Development Committed on Value for Money

By Oxford Policy Management, Julian King & Associates, Verian

January 2025

This submission responds to four of the questions raised by the IDC in their call for evidence. It is submitted by three consultancies that work extensively on value for money.

Oxford Policy Management provides analytical and policy expertise across the full policy lifecycle spanning economic and social policy and governance including health, public finance, education, climate change, and public sector management. Around half of our work is for FCDO. Julian King & Associates Limited is a New Zealand based consultancy specialising in evaluation and Value for Investment capability building. Verian Group UK Ltd provides data-driven problem-solving, including evaluation and VfM assessment, to improve public policy and society in the UK and beyond.

There is no requirement for this submission to be anonymous or treated as confidential.

Q: How does the FCDO currently define the term Value for Money? Are there any other aspects of Value for Money that the FCDO should be considering in its assessment?

• FCDO has defined the Value for Money (VfM) of its programme spend as making 'the best possible use of our resources to maximise our impact on poor people's lives' (DFID, 2020: 1). This is considered at both the overall portfolio and the programme level.
• FCDO operationalises its analysis of VfM under The ‘5 Es’: Economy – Are we (or our agents) buying inputs of the appropriate quality at the right price?); Efficiency – How well are we (or our agents) converting inputs into outputs? (‘Spending well’); Effectiveness – How well are the outputs produced by an intervention having the intended effect? (‘Spending wisely’); Equity – How fairly are the benefits distributed? To what extent will we reach marginalised groups? (“spending fairly”); and Cost-effectiveness – What is the intervention’s ultimate impact on poverty reduction, relative to the inputs that we or our agents invest in it?
• This represents quite a broad understanding of VfM, compared with other government departments which often focus on narrower economic approaches. FCDO is recognised as a leader in this respect.
• Nevertheless, we believe it could be strengthened in a number of ways. They are:
• Expanding the criteria FCDO considers relevant to VfM beyond the 5Es to other aspects that are also relevant to VfM assessment in some contexts, such as relevance and sustainability.
• Basing its assessment of VfM more systematically on a clear specification of the programme/portfolio overall value proposition and an understanding of how that value is created in a given context
• Recognising that different stakeholders may value different aspects of a programme/investment differently and taking an approach to assessing VfM which is able to incorporate different perspectives and give a voice to all relevant parties.
• Recognising that programmes often have multiple, hard-to-measure and hard-to-value intended outcomes, such as improvements in governance or female empowerment. Quantitative and money-metric measures will therefore often provide an incomplete and inadequate basis for assessing VfM. FCDO should expand the scope of information used to include qualitative data of various types (in contrast to the more narrowly focussed VFM metrics recommended in (DFID, 2020), for example).

• Some of these issues were identified in the ICAI 2018 report on VfM.
• All these issues are addressed in our VfM approach, detailed below.

Q: How effective is the FCDO at monitoring the delivery and outputs of its programming to ensure its achieving Value for Money? How could the FCDO improve its oversight mechanisms to ensure Value for Money of its ODA budget?

• In our experience, FCDO considers VfM reasonably systematically throughout the project cycle. It is analysed in business cases and reported on as part of annual programme review and reporting processes. It also commissions evaluations that include VfM assessments. However, this is not always undertaken using a single, comprehensive framework which tracks VFM consistently over the life of a given programmes.
• There are opportunities to better connect business cases, monitoring, evaluation, and VfM. For example, ex-ante appraisals should detail planned M&E arrangements integrated with programme design, with scoping of data collection requirements from the outset. This could be supported through earlier engagement of consultants or ensuring in-house capacity is available. This would better enable ex-post evaluations and VfM assessments to track impacts and benefits realisation, including revisiting assumptions from the business case.
• There are opportunities to consider VfM more systemically and at multiple levels. VfM assessments are usually undertaken at individual programme level rather than considering the collective VfM of country portfolios and the collective VfM of different programmes which may be funded by multiple donors. By examining aggregate performance, FCDO could better understand trade-offs, duplication, complementarity, and the aggregate impact of multiple interventions. This could lead to better decision-making about how to invest resources at country-level or across thematic areas.
• FCDO often channels significant resources through global partnerships and multi-donor trust funds. These arrangements can provide economies of scale, expand reach, and enhance collective impact. However, they also reduce FCDO’s direct oversight and may limit its ability to ensure rigorous VfM. Given that FCDO’s approach to VfM is generally more robust than that of many other funders, it is well-positioned to lead by example. By systematically assessing the VfM of its contributions and sharing evidence-based insights, FCDO can influence investment decisions and encourage other donors to undertake more systematic assessments of VfM.
• VfM assessments are often primarily accountability-driven with a dynamic that is sometimes characterised by defensiveness and a perceived need to “prove” VfM in order to support a business case or protect continuity of funding. This can encourage a risk-averse culture which hinders adaptation and learning that could improve VfM over time. It would be desirable to foster an environment in which VfM is learning- and improvement-oriented and where it is perceived as ‘safe’ to have honest and open conversations about VfM.
• The limitations outlined above under the preceding question also apply.
• The parties submitting this response recommend FCDO use an approach to VfM which addresses many of these concerns.
• This approach is documented in OPM/King guidelines at: https://www.opml.co.uk/publications/assessing-value-for-money. Further information is given in Verian/King guidelines, which focus on the way in which this approach can complement and broaden economic analysis: https://www.veriangroup.com/news-and-insights/guide-evaluation-value-for-money. Additional, extensive resources and discussion on the topic can be found at: www.julianking.co.nz.

The King/OPM approach

•  This innovative approach can provide a clear and systematic assessment of programmes against FCDO’s 5Es but can also incorporate and address a wide range of other criteria. It is highly flexible, able to assess VfM for a wide variety of policies and programmes, including those with difficult-to-measure and difficult-to-value intended outcomes  – such as strengthening governance.
• It can incorporate conventional economic analyses such as cost-effectiveness and cost-benefit analysis within a wider framework, but also provide an alternative when they are not suitable.

Figure 1: Summary VfM findings for a formative evaluation of the African Risk Capacity

Source: based on OPM’s ARC second formative evaluation report

• The approach provides clear, transparent and well-evidenced VfM judgements, reporting on performance against explicit, agreed, programme-specific sub-criteria and standards. Users can easily engage with the high-level summary (as illustrated in Figure 1) but can also examine the detailed evidence behind it.
• It is based around a standardised 8-step process (see Figure 2), with four defining features:
• Interdisciplinary  – combining strengths from both evaluation and economic approaches
• Use of explicit evaluative reasoning  – making an assessment of evidence against agreed standards of performance, which are expressed as rubrics
• Mixed methods  – drawing on the variety of data sources and methods that are best placed to inform VfM judgements
• Participatory  – involving key stakeholders in defining the programme’s value proposition and the performance standards against which it should be judged, as well as interpreting the evidence to support VfM judgements.

Figure 2: OPM’s VFM approach

Source: https://www.opml.co.uk/publications/assessing-value-for-money

• The approach has been applied successfully and extensively by OPM to many FCDO programmes across a range of sectors. This includes assessments covering education, female economic empowerment, climate resilience, governance, public finance, financial sector deepening and trade. It has been used in some cases as part of routine programme management and MEL – for example, in a programme to strengthen disaster risk management in Ethiopia (BRE-final report annexes, Annex I) . In other cases it has been used to undertake a ‘stand-alone’ VfM assessment as part of a programme evaluation  – such as in the evaluation of the work of the African Risk Capacity, a specialised agency of the African Union that provides capacity building and insurance for African governments to deal with climate disasters (ARC second formative evaluation report; ARC Policy Brief  – VFM). This evaluation was used by FCDO to help strengthen accountability around VfM (FCDO annual evaluation report 2022-23, case studies). All recommendations were accepted by ARC and form the basis of an action plan currently being implemented by the organisation.

• The approach is most valuable when it is integrated with programme management, routine MEL processes and wider evaluations. This ensures efficiency and consistency in the work. The VFM framework should be developed at the beginning of a programme – ideally starting with the business case, although further revisions would be required as stakeholders become involved during inception. However, it can also be used to undertake a stand-alone VfM assessment at any point during the project cycle.
• It can be used at multiple levels  – from projects or programmes to whole portfolios  – and can be scoped proportionately to the programmatic investment. The process remains the same while the criteria, standards, and types of evidence are tailored to context. High-level definitions of performance levels (poor-adequate-good-excellent) can provide comparability across a wide range of programme types.
• It has sufficient flexibility that it could also be used to assess the VfM of non-programmatic activities such as diplomatic influencing.
• The approach is recommended by the Evaluation Task Force of the Cabinet Office in its Evaluation Academy, which through direct training and training-of-trainers in the approach is estimated to have reached around 2,000 civil servants. It is beginning to be used widely across HMG in a range of MDAs.
• The approach has also been used by other consulting firms, other government agencies, UN agencies, NGOs, and foundations internationally.

Other considerations

• Whatever approach is taken to assessing VfM, there is value in ensuring that a breadth of disciplines is involved in its consideration: economists, evaluators, procurement, as well as the sector specialists who design and oversee programmes and senior management with responsibility for oversight of country and global programming.
• When VfM assessments are commissioned externally by FCDO – either as part of delivering a wider programme, a MEL project or evaluation or as a stand-alone assessment – it is important to budget sufficient resources for the work and to ensure commissioners have enough understanding to enable informed commissioning and for reviewing the work.

References

DFID (2020) ‘DFID’s Approach to Value for Money—Guidance for External Partners’, June, Finance and Performance Department, UK Department for International Development [online].

6