Written evidence submitted by UKRI (RRE0090)

1.      The UK has a world leading research and innovation system and is respected by international partners as a place where high quality, trustworthy research is undertaken. Researchers’ routine testing and replication of prior work contributes to these achievements.

2.      The terms of reference for this inquiry are set around a problem statement that outlines a crisis in the sector on reproducibility. The extent to which this problem exists in the sector is explored in Q1. The inquiry brings together several distinct issues under the broad theme of reproducibility but it is worth noting some of the differences within similar terms.

3.      There are a number of key documents that include useful definitions for this inquiry, including the US National Academies of Sciences, Engineering and Medicine (NASEM) report on Reproducibility and Replicability in Science[1]. It defines reproducibility as obtaining the same results having used the same (original) data, code, methods and analysis. Replicability is defined as obtaining the same results but using new data to answer the same scientific question. By these definitions, often when the term ‘reproducibility’ is used it is actually referring to replicability and there is an important distinction as noted above.

4.      While there are often overlaps between these issues, they are discrete terms that relate to the differing concepts – including what type of solutions or interventions should best address issues associated with them. Given their distinct, but shared importance, our response will clearly outline if we are referring to reproducibility, replicability, or both and will mostly focus on replicability which is the main issue discussed in the “reproducibility crisis”.

5.      Extremely high-quality research performed with high integrity may give results that cannot be replicated but this does not make the research any less valuable or robust. This is more common in some disciplines than in others. Lack of replication can be for positive, interesting reasons, for neutral technical reasons, or negative reasons such as error or poor research practice. Likewise, research does not necessarily rely on the replication of individual findings. Multiple, but distinct, investigations that all support the same theory or hypothesis can often provide stronger evidence than single investigations that have been replicated.

6.      An inability to replicate a research result can occur in many circumstances, for example:

1. it is impossible to recreate a unique event, for example specific geo-political changes or unusual weather events;

2. replication is so technically demanding getting any results at all is rare;

3. there is inherent variability in the system under investigation, for example cell lines grow differently in different labs;

4. there are honest mistakes in the research or its reporting, for example, tube mislabelling or a typographical error in a paper;

5. the research is poor quality, or it is poorly reported, for example, poor experimental design or inadequate methods description;

6. there is intentional research misconduct, including within research practices (for example, manipulating statistical results) or falsification or fabrication of data.

7.      These circumstances illustrate that there are clear reasons why research results may not be replicated that are not indicative of any problems in the research system. It is important that this is recognised because incentives in the system must support and welcome healthy debate about this type of lack of replication. If lack of replication is wrongly always elided with poor practice, the important process of open and engaged debate over conflicting results will be compromised.

8.      Nonetheless, in answering the committee’s questions our response focuses mostly on issues related to poor quality research or intentional research misconduct as these are areas that need addressing and minimising across all disciplines. As part of our response, we outline some of the measures UKRI and the sector are taking to tackle the issues identified.

9.      Public trust in science is an additional, important issue that has been particularly evident during the COVID-19 pandemic. The pandemic has revealed public appetite for understanding and engaging with issues of research reliability and uncertainty, and researchers have been a positive part of this public discourse. There is now both a responsibility and an opportunity to enable an informed discussion with the public on this issue. We would be happy to update the Committee on UKRI’s public engagement strategy.

10.  We look forward to working with the committee to identify and discuss how best to support the research sector and minimise factors that contribute to inadequate research practice or act as barriers to avoidable issues aroundreproducibility and replicability.

Q1: The breadth of the reproducibility crisis and what research areas it is most prevalent in

11.  The evidence base for levels of reproducibility and replicability continues to be relatively weak and incomplete. It is likely to remain so, because of methodological challenges, and because of the high cost of developing a near-complete and robust evidence base.

12.  There have been some recent studies on the extent of the problem that have taken a set of experiments and tried to replicate them which gives us good insight into how easy this might be to do. Other evidence includes John Ioannidis’ 2005 paper “Why most published research findings are false”[2], which outlined statistical and other factors that led him to that (contested) conclusion. The 2019 consensus report on computational reproducibility and replicability from NASEM (noted above) also highlights problems in the system, identifying over a dozen ways of mitigating unhelpful sources of non-reproducibility and non-replicability including training and mentoring, repeating experiments before publication, and rigorous peer review. Some of these recommendations map across to the research sector in the UK, for example, those around incentives and ‘high impact’ publications.

13.  UKRI is continuing to advance our understanding of the pressures under which the system is currently operating, including through a landscape study of research integrity, commissioned by UKRI and published in June 2020, which identified a number of factors that can affect research integrity generally, and therefore can contribute to poor reproducibility and replicability[3]. Alongside explicit studies, a July 2021 Dutch survey of around 6800 researchers found that over half of respondents reported engaging in questionable research practices[4], though these were not classed as research misconduct.

14.  These, and other surveys, have highlighted misaligned incentives, underpowered or otherwise questionable study design, wide flexibility in choices over methodology and analysis, and challenges in improving the culture, transparency and openness in research. Many, but not all, of these challenges can be addressed[5] but solutions need to respect disciplinary differences[6]. Whilst these surveys are helpful in identifying where those in the system feel that the problems are, they are not a substitute for explicit studies. As noted above, this means we have a relatively weak and incomplete evidence base on this issue. 

15.  We would be pleased to provide further details about UKRI’s research culture and environment programme which is addressing issues related to systemic pressures and unintended consequences from current incentives in the research system, and how to embed changes that improve the research environment and support and incentivise high quality research practice. Some actions we are taking in this space are outlined below.

16.  The UK Committee on Research Integrity (UK CORI) (see Q5) will have a role in improving the evidence base on research integrity. In preparation for that, we will shortly engage with the sector, publishers and international experts to explore indicators of levels of research integrity including, where relevant, reproducibility and replicability. We would be happy to share with the committee our learning from international partners and best practice to improve the evidence base.

17.  Challenges relating to reproducibility and replicability vary depending on the discipline. The visibility of a lack of reproducible and replicable research results is perhaps highest in the biomedical, health and quantitative social sciences where there may be biological reasons why an experiment cannot be replicated. In biomedical sciences, the Academy of Medical Sciences, in partnership with the Medical Research Council (MRC), the Biotechnology and Biological Sciences Research Council (BBSRC) and Wellcome, held a symposium to explore the challenges and opportunities for improving the reproducibility and reliability of biomedical research in the UK[7]. Pre-registration of clinical trials is a useful tool in dealing with these issues, which is a requirement of UKRI funded trials[8]. The Medical Research Council (MRC) runs a regular review32 to check compliance with UKRI policy33 which includes this requirement.

Q2: The issues in academia that have led to the reproducibility crisis

18.  Evidence of reproducibility and replicability issues in research are seen in academia, though that is not to say this is the only place they occur. For example, pressures in the private sector, such as commercial incentives, can also incentivise poor practice. In academia, three interconnected areas are of particular relevance:

1. Research practices, including a lack of support for open science and data sharing.

2. Research culture, including insufficient value afforded to elements beyond research itself.

3. Incentives that support poor research practices and research culture, for example, in failing to recognise the importance of negative or null results.

19.  A positive, inclusive, open research culture is key to improving issues around reproducibility and replicability and whilst we are making progress on this, there is always more to do to imbed this in business as usual practices.

20.  It is internationally[9] recognised that the research and innovation system needs to encourage the use of open data, code and methods. Ensuring that research outputs generated throughout the lifecycle of research are freely accessible to all and under conditions that enable them to be reused and built upon, can enable better scrutiny and efficiency. The UK government’s R&D Roadmap and R&D People and Culture Strategy also recognise the role of open research. UKRI action in open research and data is outlined below.

21.  Competition permeates the R&I system from how funding through both arms of the dual support system is allocated, to career opportunities for individual researchers. Safeguarding the benefits of competition while ensuring that the rules for winning support high quality research practice is necessary to counteract unintended consequences of current publishing and funding models, which have been noted by the research community as a long-standing pressure that can compromise research quality.

22.  Those in academia also note pressures due to a high workload, with many combining research with teaching and diverse service roles. A strong connection between teaching and research benefits both, and a high-quality research system requires researchers to contribute to a wide range of tasks, such as refereeing, organising conferences and providing advice to diverse stakeholders. These other important aspects of academic careers should be seen as equally as valuable to the R&I system and celebrated alongside research, with expectations for research outputs taking into account the wider roles and responsibilities of researchers.

23.  A particularly significant contributor to poor research culture and research practices is disproportionate value given to publishing in ‘high impact’ journals. In many fields, publication of negative and null results is not incentivised because of both the opportunity costs of the time taken to bring these results to publication and the low value given to these results in most researcher assessment criteria. The lack of availability of these results creates a bias in the publication record in favour of positive results, some of which will not be replicable for the reasons listed above. The disproportionate focus on high impact positive results can also lead to corners being cut in an attempt to achieve publication in a journal of choice. If we are to improve reproducibility and replicability of results in research, addressing these two consequences of current publication incentives is essential.

24.  One step being taken on this issue is that UKRI, with other funders, is adopting the Resume for Researchers format; an inclusive, narrative form CV which will be used across UKRI for researcher assessment[10]. The narrative format, with the opportunity to provide a wide range of evidence of research output, along with a requirement to provide evidence of contributions to research beyond the direct outputs of the research itself is one tool funders are using to recognise the full range of activities essential for research excellence.

25.  Recognising the relationship between research assessment and research integrity, UKRI also supports the San Francisco declaration on research assessment[11] (DORA) and has embedded this in activities, including guidance for those reviewing grant applications.

26.  Through active engagement across the R&I sector, UKRI is working with stakeholders nationally and internationally to tackle bullying and harassment. Specifically, the Forum for Tackling Bullying and Harassment brings together funding, policy and regulatory organisations who work to effect system level change through sharing knowledge and best practice to tackle bullying and harassment in all research settings. Through this forum, a cross-sector approach to discussing solutions to some of the most pervasive issues impacting research culture is expected to allow lasting change.

27.  There is a groundswell of support for improving research culture amongst the next generation of research leaders. UKRI is using the UKRI Future Leaders Fellows scheme and our Early Career Researcher Network[12] to encourage and support 'agents of change' who can promote a healthy, safe and welcoming, environment.

28.  We encourage the committee to use a wide lens as it analyses issues across the research system, and call attention to incentivising the wider research system to adopt research culture reforms that:

1. Value high quality technical and administrative support;
2. Support a positive attitude to correcting errors in literature and data;
3. Reward and value the full range of activities beyond research itself that researchers need to do to support high quality research;
4. Support an open, collaborative research culture.

Q3: The role of the following in addressing the reproducibility crisis:

1. research funders, including public funding bodies;
2. research institutions and groups;
3. individual researchers;
4. publishers; and
5. Governments and the need for a unilateral response / action.

Q4: what policies or schemes could have a positive impact on academia’s approach to reproducible research

29.  All parts of the research system outlined in Q3 can individually use specific levers to address issues related to reproducibility and replicability. For example, the Government, via the Department of Business, Energy and Industrial Strategy (BEIS) plays an essential enabling role and is an integral partner in the overall system, setting important strategic direction for the sector through strategic policy documents, such as the People and Culture Strategy, the R&D Roadmap and the Innovation Strategy.

30.  However, lasting change requires the whole sector to work together to create and embed systemic changes and this must involve those at all levels. Long standing work is underway in the sector to promote a positive research culture which includes ensuring that researchers have access to high quality resources, infrastructures and networks which would improve the quality of research. UKRI has a role to play in joining up and aligning efforts so as not to displace issues to other places in the system.

31.  Expectations of the relevant roles of the groups outlined in Q3 are included in international statements, for example those of the World Conference on Research Integrity[13], the Committee on Publication Ethics (COPE), and national concordats and agreements, such as the Concordat to Support Research Integrity[14], the Researcher Career Development Concordat[15] and the Concordat on Open Research Data[16]. UKRI are signatories of these statements and concordats.

32.  UKRI, UUK and Wellcome are currently collecting evidence of the effects of some of these statements and concordats on research culture and environment[17], working closely with the UK government[18], with a view to improving how expectations are set and met, increasing efficiency, and identifying potential gaps in current structures. We will be publishing interim findings from this work in the autumn, which can be shared with the committee.

33.  Another area that can have a positive impact on reproducible and replicable research is meta-research, or research on research. This can help us deal more effectively with reproducibility and replicability issues and can be used to help us understand some of the issues involved. There are active discussions within the research community on how meta-research should best be pursued. Meta-research may be able to identify where biases are happening and provide evidence on the potential to translate approaches between disciplines and fields.

34.  As a major research funder, UKRI has a crucial role to play in addressing the issues outlined in our response and in setting expectations for a positive research culture. Through UKRI policies, tools and frameworks we have set out expectations for both how we operate and how we expect those we fund to operate. Outlined below are just a few examples of actions UKRI is taking not just to have a positive impact on reproducible and replicable research but also the system overall.

1. We are influencing both the national and international agenda, with UKRI playing a leading role in the Global Research Council (GRC). In May 2021, UKRI, the UK Forum for Responsible Research Metrics and the National Research Foundation of South Africa published a GRC responsible research assessment report[19]. The report and subsequent Call to Action[20] aims to enable funding bodies around the world to improve assessment approaches to support a fairer, more inclusive and thriving research culture. We encourage the committee to explore how collaboration with our international partners drives forward the dialogue and implementation of responsible research assessment. 

2. UKRI is developing the Good Practice Exchange announced in the government’s R&D People and Culture Strategy. This intersectoral and international exchange forum will share information on the development, testing, and evaluation of interventions and initiatives to accelerate culture change.

3. As previously mentioned, one of the reasons for an inability to reproduce and replicate research results is because of poor research design. The MRC-NIHR Better Methods, Better Research programme[21] is an example of a UKRI initiative working to mitigate this issue. The programme provides funding to support the development and dissemination of robust research methodologies underpinning health research, improving uptake of better research methods and improving access.

4. UKRI funds projects that incorporate reproducibility and replicability as a normal part of research, and that develop and improve methodology[22]. Information on reproducibility, replicability and statistical design in grant applications is an important element. One example is the National Centre for the Replacement, Refinement & Reduction of Animals in Research (NC3Rs) experimental design hub which provides resources and guidance to aid researchers in designing and reporting research involving animals[23].

35.  As noted above, open research is a crucial element of dealing with issues relating to reproducibility and replicability. To address this, UKRI published its new open access policy[24] in August 2021, which requires research publications arising from the research it funds to be freely accessible to the public and under conditions that allow reuse. The policy clearly articulates the requirement for data access statements in research papers and highlights the need for research data to be managed so it remains “as open as possible and as closed as necessary”. The policy builds on our existing open access requirements and helps to incentivise an open research culture. Following the development of a new UKRI open access policy, UKRI’s open research priority is to work to address requirements for open research data working across UKRI and with stakeholders, building on good practice and recommendations such as the Open Research Data Task Force[25] that reported to the Minister in 2018.

36.  Beyond this, UKRI is committed to open research data as underpinning research integrity. UKRI’s research councils have research data management policies that mandate and encourage deposition and curation of research data that underpin research findings and for some time the councils have led practice in requiring data management plans. Open research data requires curation, which in turn requires appropriate infrastructure and skilled people. There are a number of infrastructure investments that enable open data in a range of disciplines, for example the UK Data Service[26] funded by the Economic and Social Research Council (ESRC) and the Natural Environment Research Council (NERC) data centres. The Research Excellence Framework, which Research England and the UK Higher Education funding bodies oversee, encourages open research data to be part of university research environments[27].

37.  Nevertheless, as discussed above, wider cultural change is required to address pressures that can lead to compromises in research quality. To help to address this UKRI is working in partnership with other stakeholders, including via the G7 Open Science Working Group[28], to identify how to reward sharing of data, software, code and other research resources to help achieve increasingly robust and reliable research outcomes.  

38.  Alongside policy, infrastructure and incentives, the development of skills and best practice is important to embed more widely open research, ensuring it becomes part of established research practice, thereby supporting reproducibility and replicability. The UK Reproducibility Network (UKRN) is proving to be a successful model in this regard contributing to more widespread engagement in the sharing and take-up of good practice across disciplines and research performing organisations. Through its Development Fund, Research England is funding the UK Reproducibility Network to accelerate the uptake of high-quality open research practices[29]. The project represents a major strategic investment by Research England that is intended to ensure the UK remains at the forefront of the open research agenda and continues to generate globally leading research of the highest quality.

39.  Opening research across the whole research lifecycle can aid reproducibility and replicability, as well as help a wider range of research activities and outputs to be recognised as integral to the transparency and practice of research. There is more exploration and analysis required to identify how and where to promote openness. There are, however, some established cultures and practices in the use of preprints and registered reports, for example in medical research, and recently we have seen their importance in research undertaken in response to the COVID-19 pandemic.

40.  Research England is funding the development of Octopus[30], a new primary research record intended to enable transparent sharing of research across its whole lifecycle from problem to hypothesis or rationale to data or results and then through to analysis and implementation. This is in development but reflects the emerging innovations required to support more open research processes. UKRI is working with Wellcome, the Gates Foundation and stakeholders around the world to examine and learn lessons from research undertaken in response to the current pandemic in a joint study gathering evidence regarding data sharing and wider open research practice to better inform us in undertaking transparent research and responding to global challenges. We would be pleased to share further details of this work with the Committee.

Q5: How establishing a national committee on research integrity under UKRI could impact the reproducibility crisis.

41.  The UK Committee on Research Integrity (UK CORI) will promote research integrity across the UK and internationally. As recognised in the previous House of Commons Science and Technology committee’s report on research integrity[31] there are many threats to the integrity of the research record: from honest errors or publication bias to deliberate misconduct. UK CORI will look at research integrity in the broadest sense. This will incorporate reproducibility and replicability but set this in the context of related issues and pressures which cause problems beyond and not always related to reproducibility and replicability.

42.  It remains the case that efforts to improve research integrity, and reproducibility and replicability, are hampered by a lack of evidence about the scale of the problem, and the nature of the problem across different disciplines. UK CORI will help to build and communicate that evidence and from this, identify effective solutions. UK CORI will publish an annual state of the nation report establishing baselines in the first years which will allow for evaluation of progress in future. Identifying what the right measures are and which trends to monitor will be a key output from UK CORI.

43.  In recruiting the inaugural chair and members for UK CORI, we are seeking individuals with experience of championing change. UK CORI will be drawing on, and amplifying, the positive actions and initiatives that already exist in the sector.

44.  Through a focus on systemic pressures that impact individual behaviours, including the levers and drivers of certain sectors like funding and publishing, the committee will work to identify incentives and opportunities to improve integrity. By offering leadership and evidence on how to address poor practice through improved training, better support, decreased pressure, and improving culture, UK CORI will be working on the issues that influence research quality, including reproducibility.

45.  As a cross-disciplinary group, UK CORI will be well placed to encourage discussion and share learning between different areas of the research sector. It will be a high-profile addition to the sector, able to convene research organisations, funders and publishers to share best practice and common approaches which will support high integrity across the entire research lifecycle. The committee will work to support and create a research environment where discussions and debate about the principles of research integrity can be considered in a safe, positive environment leading to a culture and practice of continual improvement for the sector. As a national group, UK CORI will provide the UK with leadership on the global stage, as issues of integrity cross borders and the most effective solutions will involve the setting of international expectations and practices. We look forward to sharing insight from UK CORI with the committee, the sector and the government to ensure we are learning from their work.

About UKRI

Launched in April 2018, UKRI is a non-departmental public body sponsored by the Department for Business, Energy and Industrial Strategy (BEIS). Operating across the whole of the UK with a combined budget of more than £7.9 billion (2021-22), UK Research and Innovation (UKRI) brings together the seven disciplinary research councils, Innovate UK and Research England. Our vision is for an outstanding research and innovation system in the UK that gives everyone the opportunity to contribute and to benefit, enriching lives locally, nationally and internationally. Our mission is to convene, catalyse and invest in close collaboration with others to build a thriving inclusive research and innovation system that connects discovery to prosperity and public good.

October 2021

10