The breadth of the reproducibility crisis and what research areas it is most prevalent in

4 There has been increasing discussion of reproducibility in research in recent years: its role in research, whether there is a ‘reproducibility crisis’ and, if so, the extent and breadth of such a crisis. There is concern amongst many researchers about levels of reproducibility in research [4]. There is also debate about the extent of the ‘reproducibility crisis’ and how best to respond to reported problems [5] [6] [7].

5 Concerns about reproducibility are part of an increasing interest in issues of research integrity in general – for example, both enquiries [8] and subscriptions [9] to UKRIO have increased year-on-year. There has also been an increase in public and media interest in the reliability of research, particularly in light of the ongoing pandemic [10].

6 Concerns about reproducibility have been increasingly linked to wider issues in research, such as research culture [11] [12] and, in particular, the incentives which act on the research community, some in a positive way and others in a negative way [13]. Drawing from our own work and wider discussions of problems with research culture, key drivers of irreproducibility would appear to be:

Competition.
Research culture.
Focus on publications as key metric for assessing and promoting researchers.
Lack of security in research career structure and lack of clear career pathways.
Focus on novelty and impact over and above good research practice and reproducibility.
Time pressures on researchers.
Embedding and exacerbation of the above issues through how research is assessed.

7 Based on our advisory work with researchers and institutions, it is better to take a step back and look at research integrity as a whole, rather than consider reproducibility in isolation from other matters. In a study on incentives carried out by Vitae, UKRIO and UKRN in 2020, 90% of survey respondents agreed that research integrity is compromised at least some of the time [14]. There has been research showing that research that is not reproducible is less likely to be true [15] and that a lack of replicability can be linked to biases in the research [16], so a relationship can be seen between reproducibility and good research practice.

8 However it should be noted that research is, by its nature, an exploration of the unknown and as such it is never guaranteed that research should be reproducible. Research is about iterative testing of hypotheses so much research will eventually be shown to be inaccurate. In addition, some kinds of research are inherently not reproducible.

9 We have not taken a policy position on the precise extent and breadth of issues of reproducibility in research, as we recognise this is a topic of debate. However, we have embedded this topic in our work: via our virtual events programme [17], our work on research culture, and including transparency and replicability in our published guidance [18]. As an independent and confidential advisory body, we pick up on growing concerns about reproducibility emerging from researchers and organisations, often linked to broader concerns/ factors relating to research culture [19] [20] [21].

10 What is important for all research is that it meets standards of good practice: that it is high quality and of high ethical standards. Open research and reproducibility of data is an important aspect of this. A landscape study showed that 89% of survey respondents thought that sharing research methods was important for research integrity and 79% believed sharing data was important for research integrity [22].

The issues in academia that have led to the reproducibility crisis

11 Problems with reproducibility are a symptom of broader cultural issues in research, issues which have a practical impact on researchers and on research, in particular its quality and ethical standards.

12 ‘Research culture’ is influenced by a variety of factors and incentives [23] [24] [25] [26]. Such factors include, but are not limited to: assessment of research proposals; how research is funded, managed, monitored, disseminated and assessed; how researchers are recruited, assessed and promoted; workload models; equality, diversity and inclusion; staff development; staff welfare and the impact of bullying and harassment; leadership of research teams and research organisations; and the impact, whether positive or negative, of incentives in research, such as those from funders and employers.

13 Addressing negative aspects of research culture requires long-term work from all sections of the research community (though short-term interventions at the local level will also have a positive effect [27]) and leadership and self-reflection from those in senior positions [28] to help create a research culture that encourages open research practices.

14 Studies have shown that researchers generally aspire to do rigorous and ethical work, but there are major concerns about pressures to focus on and report positive results, and also perverse incentives to fabricate, alter, omit or manipulate data [29] [30] [31]. Open research practices help to reduce such problems, for example through researchers being transparent with their raw data.

15 Aspects of research culture, such as competition, “are believed both to bring out the best in people and to create incentives for poor quality research practices” [32]. Based on our experience of advising researchers and institutions, UKRIO’s view is that more work must be done to mitigate the negative impact that some incentives can have.

The role of the following in addressing the reproducibility crisis:

16 Research funders, including public funding bodies: funders should continue to emphasise openness and transparency, and the career development, working practices and infrastructure needed to support this. They must take strong action on those incentives which have a negative impact on reproducibility and research integrity.

17 Funders have great power to effect cultural change through their role in the research ecosystem. But funding bodies and those affiliated with them must take care not to inadvertently introduce new negative incentives or bureaucracy when taking action to address current problems. In UKRIO’s experience and that of others, the status of major stakeholder organisations can sometimes lead to misperceptions, with guidance and initiatives from such bodies being viewed as things which must be dogmatically adhered to, instead of their intended use as aids to ongoing reflection and improvement.

18 Research institutions and groups: as researchers consistently report that pressures in the system impact negatively on research quality and reproducibility [33] [34] [35], employers should adopt working practices which reduce pressures and promote research integrity, including open research and high ethical standards. However, many pressures are manifestations of the research system in general and cannot be addressed by research organisations alone; employers must be supported by funders, publishers, Governments and specialist bodies such as UKRIO.

19 The House of Commons Science and Technology Committee’s 2018 report into research integrity [36] noted problems with the current system of self-regulation of investigations of research misconduct. Concerns about research misconduct remain and must be addressed by the research community. UKRIO is ready to lend its expertise and unique perspective to this work.

20 Individual researchers: various studies show [37] [38] [39] that researchers feel primarily responsible for good research practice, but feel constrained by pressures in the system. The right environment needs to be in place to facilitate greater research integrity and enable them to produce high quality and reproducible research.

21 Researchers also have power to effect change, as members of REF panels or those for ethical review, peer review or grant review; and as mentors of colleagues. Activities by individuals and peer-led groups can be effective [40] [41]. However, for researchers to be able to engage more with issues of research integrity and reproducibility, they need to be allowed time to do so [42] [43] [44].

22 Publishers need to put systems in place to require transparency of data and this in turn would encourage open research practices if it was a requirement of submission. Registered reports are an example of a significant movement by publishers that will improve reproducibility, transparency, and self-correction. The ability to publish replication studies (not new research) and null results is also equally important in the drive for rigour and reproducibility, and all types of organisation need to work collectively to encourage this.

23 The problem of unscrupulous/ ‘predatory’ journals continues and more needs to be done to address this. Such journals encourage and provide a route for substandard and fraudulent research.

24 Clearer guidance for peer reviewers on how to review, and respond to, matters of reproducibility would be helpful. Reproducibility is obviously primarily down to how the research is planned, carried out and reported, but scrutiny during peer review would be useful.

25 Governments and the need for a unilateral response / action: Governments must take responsibility for ensuring that public funding for research is well spent. They also need to, as far as possible, measure progress and hold institutions and funders to account for ensuring that their self-regulating processes are up to scratch.

26 Governments must promote practices which incentivise open, transparent, ethical and reliable research. They must avoid inadvertently incentivising poor practice.

27 Governments should adopt best practices for research employers and funding agencies, both for the positive impact this would have on the research that they carry out and fund and also as an exemplar for the research community. They should provide support to their researchers to engage in open and transparent research, and encourage organisations from which they commission research to do the same.

28 A wide variety of initiatives and organisations are already working to support and enhance research integrity and reproducibility, there should be an emphasis by Governments on avoiding duplication of effort. Existing expertise and activities can be enhanced through their promotion and/or support by Governments, enabling existing good practice to be shared more widely and then built upon further.

What policies or schemes could have a positive impact on academia’s approach to reproducible research

29 Research is a complex ecosystem and changes across the board are needed: ‘support at the individual researcher level is unlikely to be sufficient in embedding a culture of research integrity… it is clear that no one stakeholder, group or individual can make significant changes in isolation’[45]. This would be long-term work: ‘[a] sustained, multi-stakeholder effort that is likely to ensure the UK has a world-leading research ecosystem underpinned by the highest levels of research integrity’. [46]

30 As part of wider change, smaller interventions can also be effective [47], especially with strong leadership [48], and there are many good examples of how organisations can encourage responsible research practices (see our recent webinar on Recognising and Rewarding Responsible Research Practice for some examples [49] [50] [51]).

31 A focus by all on greater transparency would be helpful to enhance research integrity. This would be in line with the UK Research and Development Roadmap, which states [52]: ‘We will...strongly incentivise open data sharing where appropriate, so that reproducibility is enabled, and knowledge is shared and spread collaboratively..’

32 It is vital to have a clearer picture on the overall state of research integrity across all sectors of research in the UK and how the UK compares to other nations, including information on how poor practice and suspected misconduct is being addressed. This would enable more accurate assessment of current problems and of progress made in correcting them, and provide assurance to the public and policy makers. Gathering this information will require collective working from funders, research organisations, publishers and policy makers, supported by specialist bodies such as UKRIO and UKRN.

33 Improving reproducibility and research integrity will require time and effort from all sections of the research community. Two particular challenges will be ensuring a long-term commitment to cultural change from employers, funders and publishers, and ensuring that researchers are given time to engage in the open and self-reflective practice that is needed to safeguard and enhance research integrity.

34 The practicalities of research vary between sectors, disciplines, type of professional role, career stage, etc. A ‘one size fits all’ approach will only have a limited reach; initiatives and support must be tailored to the particular needs of the type and discipline of the research in question.

How establishing a national committee on research integrity under UKRI could impact the reproducibility crisis

35 UKRIO welcomes the creation of the UK Committee on Research Integrity (UK CORI)[53] and its commitment to working with UKRIO and other bodies, avoiding duplication of effort [54]. We look forward to collaborating on the issues that matter to the research community and the public.

36 The creation of UK CORI was recommended by the House of Commons Science and Technology Committee in its 2018 report into research integrity [55]. The Committee also recommended that UKRIO, as part of its continuing work, should advise on the creation of UK CORI [56] and our expertise has helped shape this new body.

37 Like the Signatories Group of The Concordat to Support Research Integrity, the UKRI-backed UK CORI will have great convening power, facilitating discussions and work by funders to address ongoing problems with incentives in research, and working with other bodies to safeguard and enhance a healthy research culture. UK CORI could also promote the work of expert organisations working on research integrity, such as UKRIO and UKRN, and the uptake of practices/initiatives which can have positive effects on reproducibility, promoting methodologies and data that are open and transparent.

38 Historically, researchers and research organisations can be wary of speaking freely with funding bodies, or with organisations perceived to be affiliated with them, about the challenges and problems that they are grappling with. As an established independent and confidential advisory body, UKRIO has often played a role in communicating anonymised summary information on these challenges and problems to funders and others. We look forward to using this aspect of our work to inform the activities of UK-CORI.

About the UK Research Integrity Office

39 UKRIO [57] is an independent charity, offering support to the public and research community to further good practice in research. We pursue these aims through a variety of means, including: our publications on research practice [58]; our education, training and awareness-raising activities [59]; the support and services we provide to organisations; using our unique experience and expertise to inform and support national [60] and international approaches to research integrity; and by providing expert guidance in response to requests for assistance. Use of our services has grown year on year, with our charity addressing over 150 formal requests for assistance in 2019 [61], as well as many informal requests.

40 Since 2006, UKRIO has provided independent, expert and confidential support across all research disciplines. We cover all research sectors: higher education, the NHS, private sector and charities. We provide advice, support and publications free at the point of delivery for researchers, organisations and the public. Research organisations can also subscribe to UKRIO to access more in-depth support to safeguard and enhance good research practice within their organisation.

41 Over 100 research organisations currently subscribe to UKRIO and we feel this shows the value of our independent and practical model of support. While nominally a UK organisation, UKRIO has global reach with its internationally applicable guidance and virtual events.