VGI0007
Written evidence submitted by Ms Sarah Coleman
(University of Manchester)
1.1 This written submission draws upon doctoral research undertaken at The University of Manchester, funded by the Economic and Social Research Council (ESRC) under the ‘Project X’ programme - a collaboration of UK government departments, academia and industry. Project X generates unique insights into the performance of major projects and programmes within and outside government through excellent 'co-produced' research.
1.2 The research described in this submission seeks to understand how organisations undertake benefits evaluation after project outputs have been delivered in order to gauge the impact and value created by the project. This evidence is crucial to understanding project impact on policy and strategy, and on investment decisions. This research is undertaken through the lens of UK government major projects. Research data was obtained through interviews with project and project-associated individuals and desk-top reviews of project documentation gathered from eight anonymised programmes from the Government Major Projects Portfolio (GMPP).
1.3 Project performance is usually assessed against a ‘triple constraint’ of time, cost and quality/scope (the ‘iron triangle’). More recently however, the focus has shifted towards project ‘effectiveness’, thus encouraging a greater appreciation of the outcomes and impact generated by a project investment. As well as creating value during delivery, projects create value once the output or ‘asset’ (be it service or tangible product) has been handed over into the operational environment, adopted and utilised.
1.4 Therefore, project benefits realisation can be considered as ex-post project investment evaluation, an important consideration in evidencing improved organisational performance and in evidencing policy impact. This research aligns benefits realisation with impact evaluation, more specifically as ‘ex-post evaluation’ defined by the NAO (2013) as ‘the activity of examining the implementation and impacts of policy interventions, to identify and assess their intended and unintended effects and costs’ so attempting to respond to the question ‘What difference has an intervention made?’. It has been previously established through research that benefit statements covering both tangible and intangible elements are a key part of investment decision-making but are fraught with challenges. The range of themes (social, economic, and environmental benefits as well as financial) identified for project business cases demonstrate the multi-faceted nature of projects and their multi-dimensional impacts.
1.5 The Well-being of Future Generations (Wales) Act (2015) makes provision requiring public bodies to consider the economic, social, environmental and cultural well-being of Wales in accordance with sustainable development principles. The Public Services (Social Value) Act (2013) requires those who commission public services to consider how wider social, economic and environmental benefits may be secured. Further, the recent HM Treasury Green Book Review emphasises wider considerations of benefits including social value, away from traditional considerations of financial benefits and highlighting indirect effects and wider economic impacts. Together, these emphasise the need for wider perspectives on benefits and evaluating impact across a long time horizon, and inform thinking around the management of project benefits.
The main conclusions from the research, and their relevance to the questions posed by the inquiry, are outlined below.
2.1 Good examples of the wider implications of project benefits including supporting healthy ecosystems, improving safety, improving regional economy, capability development (often using apprenticeship schemes), mobilising currently immature industry sectors, improving response times from weeks to minutes.
2.2 Benefits may be realised through the work of business change and through-life operations. Here, key considerations are business change supporting preparation and business readiness for handover of outputs, and the subsequent adoption and use of outputs in the operational environment.
2.3 However, government organisations can struggle to provide an evidence baseline for benefits; to decipher which benefits are particularly relevant often amid a volume of benefits provided for the business case; to assess the impact of project changes on target benefits; to provide the data and integrated software systems necessary to track evidence of actual benefits realised once outputs have been handed over to the operational environment. The integration of multiple systems and platforms to enable benefits data sharing remains problematic, particularly in the context of co-benefits.
2.4 There is awareness of the need for ex-post project benefits realisation but at the same time a certain reluctance to undertake this. There is evidence in some cases that a project ‘sponsorship’ approach can reinforce the importance of protecting benefits throughout the life of an investment.
2.5 Ex-post project benefits realisation is regarded as a project issue rather than as a ‘business’ or operational issue and, as such, it can be a struggle to get the business to take notice as well as ownership. Moreover, it is recognised that the call for ex-post project benefits realisation implies availability of budget and resources; however, it is typically difficult to access budget to allow for these activities. In some circumstances, the general practice is to provide ‘bucket codes’ for benefits realisation activities which can be undertaken in-house or out-sourced to external organisations.
2.6 Typically, those projects with robust, planned and budgeted ex-post benefits realisation activities have support at senior levels within the organisation. For example, where the role and responsibilities of Senior Responsible Owners (SROs) span across project delivery into requirements for ensuring ex-post project benefits realisation planning and action, and, further, where there is consistency of personnel within the role.
2.7 Ex-post project benefits realisation is key to placing benefits at the heart of project design and delivery because it answers the ‘so what?’ question, providing the feedback loop between target and actual impact. This research seeks to respond to this question, acknowledging that governments and organisations continue to invest in projects to deliver outcomes.
2.8 UK government organisations may experience challenges in demonstrating the realisation of those benefits in practice: ex-post evaluation is not performed systematically, and ex-post project benefits realisation is not routine to the general process of project benefits management.
2.9 There is a general presumption that ex-post project benefits realisation will occur; however, this is often without due consideration of the necessary capability, capacity, budget and timeframe for it to be undertaken. This research establishes that planning for ex-post project benefits realisation should be undertaken early during early-stage activities: there is a need to plan for and to realise the business case target benefits across project execution, handover of outputs and within the operational environment. The mindset needs to move further than simply a need to deliver outputs and evaluating project management efficiency.
2.10 Long duration projects are particularly vulnerable to the effects of time on benefits management practices. This affects the appetite to undertake ex-post project benefits realisation: a key question is “what is my ability to realise benefits several years from the project initiation stage?”, acknowledging increasingly tenuous cause (the project) and effect (the benefit). This research also uncovers concern of duplication and cannibalisation of benefits due to subsequent projects, limiting the impact of the project.
2.11 Ex-post project benefits realisation is not solely dependent on project delivery. There are many contributions including upstream and downstream boundary spanning activities between business case development and the project, and between the project and the operational environment. These include the rigour of early stage estimating and business case development, as well as business change activities at handover and within the operational environment to accommodate different working practices. Ex-post project benefits realisation is dependent on prior stages; for example, estimating and reference class forecasting, statement of assumptions, biases and heuristics during business case development. As such, it is recognised that although there are no guarantees for benefits, rolling benefits estimates and calculations across the project lifecycle (for example, using GMPP assurance reviews) is a robust method of continually evaluating target benefits prior to handover into operations and business-as-usual. Therefore, this research recommends that the role of the GMPP assurance review be acknowledged as an early warning indicator for realising project benefits; further, that as well as providing a delivery confidence assessment (DCA), the role of GMPP assurance reviews to be extended to provide benefits confidence assessment (BCA) in terms of scale and likelihood of ex-post project benefits realisation.
2.12 The NAO (2018) recommendation that the role of the IPA encompass the first 12 months of full operability of the asset within the operational environment after formal project closure to ensure ex-post project benefits realisation. There is a clear role for the IPA to monitor, measure and report on those major projects leaving the GMPP once they have delivered into the operational environment.
2.13 This research acknowledges the difficulty in establishing baseline data to evidence movement and change. In some circumstances, research projects were frustrated by benefits realisation data being fed through from multiple organisations outside their direct control, and the lack of consideration around the data and systems used to track benefits after project outputs had been implemented. It is recognised that baseline data should be available at handover between the project and the operational environment since ex-post project benefits realisation is perceived as ultimately an operational or ‘business-as-usual’ activity. Suggested solutions are common data ingest to pull data into a single database for analysis.
2.14 Some of the projects researched acknowledge that some benefits may be “more theoretical than practical”, so adhering to HMT’s (2018) ‘principle of proportionality’ whereby the approach to benefits measurement should be ‘prudent, proportionate, and appropriate’.
2.15 Ex-post project benefits realisation requires ‘proportionality’; i.e. that effort, timing and resources are required in line with actual need, so relating back to the volume and type of benefits originally specified in the business case.
In summary, this research establishes that ex-post project benefits realisation continues to be controversial. It clarifies a spectrum of strategies in use across UK government projects and establishes that current approaches to ex-post project benefits realisation are typically regarded as discretionary; that where considered, strategy and planning for ex-post project benefits realisation are typically developed at the back end of the project; that considerations around benefits data for ex-post realisation is considered late. Therefore, to be effective in evidencing change and movement ex-post project benefits realisation must be sited outside the project itself, be aligned with organisational performance and impact evaluation functions, and be proportionate to the context. Moreover, it must have a broad focus on socio-behavioural, time-horizon and environment considerations as well as the focus on the traditional, rational process and technical considerations.
His Majesty's Treasury (HMT) (2018). Guide to Developing the Project Business Case. Better Business Cases: for better outcomes. London, UK: Crown Copyright
National Audit Office (NAO) (2013). Over optimism in government projects. London, UK: The Stationery Office
National Audit Office (NAO) (2018). Projects leaving the Government Major Projects Portfolio. London, UK: The Stationery Office (HC 1620 Session 2017–2019)
March 2024