Written evidence submitted by Dr Julian Molina (PEC0004)

Introduction

• My name is Dr Julian Molina. I am a Lecturer in Public Policy at the School for Policy Studies, University of Bristol, BS8 1TZ. I am under contract with Bristol University Press to write a book about the history of criminological expertise, information systems and predictive analytics in UK and US criminal justice systems since the early 1950s. My evidence comes from research to better understand the history of MoJ prison population projection statistics.
• Between 2018 and 2021, I worked in the Civil Service as a Government Social Research professional in two arms’ length bodies of the Ministry of Justice (MoJ): the Prisons and Probation Ombudsman (2018-2019) and the Office of the Victims’ Commissioner of England & Wales (2019-2021). I hold a doctorate in Sociology from the University of Warwick and a master’s degree in the same subject awarded by Goldsmiths, University of London.

Understanding MoJ prison projection statistics

1. As far as I understand, the Home Office began seriously testing the use of computer-based microsimulation modelling of the criminal justice system during the 1980s, following attempts to develop dynamic “stock and flow” models of the US criminal justice system in the 1960s. [1]
2. The following evidence focuses on current MoJ practice. It reflects the committee’s inquiry into prison estate capacity and how data, research, and analysis inform the MoJ’s long-term strategy. In detail, my evidence concerns the MoJ's current statistical methods for preparing projection statistics, the robustness of these techniques, its use of internal/external evaluations, and the public accountability of these prison population projections.
3. In the MoJ’s Prison Population Projections 2023 to 2028, England and Wales, the report describes the methods used to produce these projections as follows: “The projections are produced using a model of flows of offenders into and out of prison, which estimates the resulting prison population each month”.[2]
4. One step I took to understand these methods better was to submit a Freedom of Information (FOI) request to the MoJ on June 11, 2024. I asked for information on what the MoJ refers to as its “prison projections modelling system”.
5. In external documents, the MoJ describes three sub-models used to prepare these statistics: the Prison Population Stock-Flow Model, the Courts and Sentencing Model, and the Demand Projections Model.[3] My FOI requested that the MoJ release information related to these models, namely instruction manuals, guidance, and in-house evaluation reports about the method and methodology of this modelling.
6. The MoJ communicated their final decision about my request on September 3, 2024. The MoJ rejected this request and did not disclose any information.[4] I appealed this decision and submitted a complaint to the Information Commissioner’s Office (ICO). The ICO accepted my complaint and is currently investigating the MoJ’s response. I expect the ICO’s final decision in January 2025. However, in response to the ICO’s investigation, the MoJ released two internal documents about their modelling and answered a series of questions posed by the ICO. A list of pertinent evidence for the committee’s inquiry is provided below.

Internal MoJ projection statistics documentation

7. In response to the ICO’s questions, the MoJ provided further information about its approach to modelling the projected demand on the prison estate. The MoJ’s modelling of the criminal justice system does not take the form of a single model but a network of models which incorporates a set of assumptions into its calculations of different parts of the system.[5]
8. This network of models seems much more complex than the “three” models described in the MoJ publications. Each model has several parts, which the MoJ refers to as “sub-models.”
9. To make changes to the MoJ’s models between regular runs of its projection, the MoJ works with policy and operational teams to understand changes in assumptions that will impact material demand.[6]
10. The MoJ’s modelling requires it to make sentencing assumptions to estimate what proportion of court demand will result in a custodial sentence.[7]
11. The MoJ uses historical ratios to take court disposal projects and convert them to convictions, sentences, and prison receptions projections.[8] A diagram provided by the MoJ is included in Annex 1.
12. The MoJ provided technical guidance for one microsimulation model for the “prison population stock-flow model”.[9] The MoJ documentation “only relates to part of the main prison stock flow doc” and not additional models or calculations used to run these projections.
13. The MoJ stated that:
1. The MoJ does not hold documentation for the other parts of the modelling. The models are run by members of the team who have received oral training on the job using the previously shared flow diagrams.[10]
14. The MoJ confirmed that it did not hold any other documentation to prepare the models:
1. That is correct, the MoJ does not hold any other documentation for the existing models.
15. The technical guidance mentioned in 12 (above) was a 6,733-word document that included technical advice on preparing the projection statistics, including sampling and resampling techniques.
16. This technical guidance was prepared in 2022 and is now out of date. As the MoJ stated in response to the ICO:
1. It was last updated in 2022, and parts are now out of date as the modelling has been slightly updated but there are no updated notes or documentation.
17. The document includes notes suggesting active causes of concern and that improvements are required. I have no reason to confirm or doubt that these concerns were addressed in more recent updates to this sub-model. In Annex 2, I quote 15 statements from this document.
18. Each of the comments in Annex 2 suggests a set of known issues with the robustness and accuracy of the model. This seems to be the only available MoJ-acknowledged paper trail for quality assurance purposes for this sub-model. In its response to the ICO, the MoJ stated that no documentation exists about its other models or sub-models.[11] It would, therefore, be reasonable to assume that there is no other documentation or paper trail about any known areas for improvement to support quality assurance processes for any of the MoJ’s other models or sub-models.
19. Furthermore, the MoJ confirmed that it had undertaken no external evaluation of its modelling techniques. Their response to the ICO read as follows: 
1. The MoJ do not hold in-house evaluation reports on the models’ methods and methodology. Each year we discuss the performance of the projection and publish this information for transparency as part of the regular publication document (see Section 3 of the last publication document here: Prison_Population_Projections_2023_to_2028.pdf)
20. The MoJ further stated:
1. As above the MoJ does not hold in-house evaluation reports on the models’ methods and methodology but we do consistently publish a discussion of the projection performance and our resultant thinking on any new assumptions.
21. I acknowledge that the MoJ is not legally obligated to commission any internal or external evaluations of its modelling methodology.

NAO assessment of MoJ population projections

22. The Committee will note that the National Audit Office (NAO) reviewed the MoJ’s capacity and demand projections in its recent report.[12] The NAO assessed how the MoJ models future demand and capacity and uses its projections (see Part 3, pp. 39-41).
23. The NAO reviewed the MoJ population modelling governance in its report:
1. We reviewed MoJ’s population and capacity modelling against our good practice framework and found evidence of good governance and appropriate testing of assumptions. MoJ has well-established oversight boards to scrutinise model assumptions, and it seeks input from relevant government departments. For example, it consults the Home Office on crime-based assumptions. We have also seen evidence of MoJ reviewing past projection performance to identify weaknesses and inform model development.[13]

24. The NAO noted, “The accuracy of MoJ’s population projections has declined slightly in the past few years”.[14]
25. The NAO recommended that the MOJ change its reporting practice for its capacity projections. As the report states:
1. To provide greater transparency to the public and parliament, MoJ should commit to routinely publishing capacity projections alongside its demand projections.[15]

Concluding remarks

26. Based on the information the MoJ released, serious questions remain unanswered about its modelling methodology's robustness, accuracy, and transparency. The NAO report did not fully address this, as it focused on governance rather than technical procedures for the MoJ’s prison population projection statistics. However, the lack of external scrutiny of the MoJ’s modelling methodology does not detract from the high-pressure and thorough work done by MoJ researchers and analysts involved in preparing these statistics.
27. However, I invite the committee to call on the MoJ to rethink its current approach to ensuring a robust quality assurance process, transparency of its modelling methodology, and its use of social research. The Government’s ambition to address a crisis in the prison estate capacity means it should consider better tools to model and understand future crises. As Lord Timpson told the Prison Governors Association Group in October 2024, the justice system has been close to grinding to a halt, but the “crisis isn’t over”. He went on, “If things don’t change, we’ll end up in the same position all over again… Sooner than we care to mention.”[16]
28. One modest and reasonable change would be to ask the MoJ to rethink its current approach to model potential futures and provide information about the future of prisons to the public and Parliament. Statistical projections must be part of this picture, but not the only part. Statistical projections do not explain why a crisis arises or the processes leading to crises. The release of prison population statistics should be further contextualised by alternative social research tools, such as scenario planning, that provide greater transparency and accountability about alternative futures for both prisons and criminal justice.
29. Reporting increasingly uncertain statistics that are declining in accuracy and are not regularly scrutinised by external technical specialists does not best support public understanding of the prison capacity crisis. Similarly, the continued use of modelling methodologies that are not routinely stress-tested by external evaluators, stakeholders, and parliament and are not transparent to the public or Parliament does not support policy and operational planning.
30. Part of the answer to this issue will require decreasing uncertainty and improving accuracy while increasing public transparency and accountability. Another part of the answer is to address today’s capacity issues with better use of data, research and analysis. There is considerable scope for commissioning internal and external social research to understand how the government could address long-term systemic issues with prison capacity and demand. This expanded use of social research would provide much-needed, high-quality information for public debate about the future of the prison estate.

END

Written evidence submitted by Dr Julian Molina (PEC0004)

Annex 1: Ministry of Justice (undated) CJS demand model documentation. Page 5.

Annex 2: Ministry of Justice (undated) ‘PrisonMicroSim Model Readme.text’.

Direct quotations from the document (all errors in the original document)

• *Note- Ideally would have improved input validation included here such as making sure InflowMonth and Value which are a must to be included.*
• *Note- currently formatting date here- does not affect if date already formatted but not very robust, improve!*
• An improvement would be to set a random day of the month for each entry- uniformly distributed throughout the month.*
• *Note- should improve column selection as some unnecessary and confusing ones are left (e.g Date_Of_Release when MicroSim_Release_Date is the model estimate)*
• *Note- Remand Scaling Factors option is not fully developed and has live issues (further, it was never used).*
• *Note- Ideally would have improved input validation included here such as making sure InflowMonth and Value which are a must to be included.*
• *When applying the receptions distribution there are large enough pools of entries in FullSample for each Band given the distribution period is as suggested- a year (if not more). Since the Bands are a key classification in prisons that has been present and defined in its current meaning for a long time, there is little reason to doubt that assumption.*
• *Note- There was further work related to reducing the replacement when sampling, specifically for dropping used discharge dist entries between iterations. This is swithced off due to the effects seen being marginal on total population while significant on runtime. It is largely encapsulated in the function `remove_used_dist_entries()`. There is a large section of comments within the function further explaining the situation and the current choice. Please see if interested.*
• Note that 15 consecutive attempts with no new match is the threshold that tends to get hit and the 500 iterations maximum is just another safety to avoid infinate loop issues.
• *Note- should improve column selection as some unnecessary and confusing ones are left (e.g Date_Of_Release when MicroSim_Release_Date is the model estimate)*
• *as mentioned above, the function currently has two practically separate flows for handling new receptions' discharges and basepop decay output. That does not have to be the case as the two aggregation procedures (separated via conditional statement) have a lot of overlap.*
• *Given it is best to think of the discharge distribution year in this case as the base year. As a to do item would be worth refactoring to reflect that in variable names and function documentation.* 
• *Note- this is an experimental feature and has not been used for projections, would need further development first*
• *There is an argument to be made for adding the basepop post processing functionality into the determinate collator function. The basepop post processing in mind is the dropping of month where basepop date is and discharges for that month. Moreover, for completeness it may be worth also bringing in the estimation of unaccounted flows from last Friday of the month (when basepop is taken) up to actual end of month.*
• the FullSample record does not necessarily have each entry represent a prisoner but rather spells in prison (an aggregation of sentences) which does not have a one to one correspondence to prisoner entires.

January 2025