Written Evidence Submitted by the UK Reproducibility Network Steering Group

(RRE0041)

Authors

Marcus Munafò, University of Bristol

Chris Chambers, Cardiff University

Alexandra Collins, Imperial College

Laura Fortunato, University of Oxford

Malcolm Macleod, University of Edinburgh

As the UKRN Steering Group.

The UKRN Steering Group works to coordinate within and across the constituent parts of the UK Reproducibility Network, including our 57 Local Networks, our 20 Institutional Members, and the 37 organisations that are part of our Stakeholder Engagement Group.

Summary

• The UK Reproducibility Network (UKRN) brings together Local Networks of researchers, Institutions, and External Stakeholders (funders, publishers, learned societies and other sectoral organisations), to coordinate action on reproducibility and work to ensure the UK retains its place as a centre for world-leading research.

• The UKRN Steering Group has coordinated responses to this inquiry from all three groupings and here we synthesise common themes across these submissions, with a particular focus on the final question – how establishing a UK Committee on Research Integrity could impact the reproducibility “crisis”

• There are many factors that contribute to the reproducibility and replicability of scientific research. Future work should focus on understanding this system, with improvements requiring efforts across all parts of the research ecosystem.

• A UK Committee on Research Integrity could play an important role in coordinating these efforts, working collaboratively with researchers, institutions, funders, publishers, learned societies and other sectoral organisations, and providing a monitoring and reporting function.

• New ways of working and emerging innovations hold a great deal of promise, but it will be important to invest in meta-research activity to ensure that these approaches are evidence based, work as intended, and do not have unintended consequences.

• Addressing reproducibility will require working collaboratively across the research ecosystem in order to share best practice and make the most effective use of resources. UKRN has begun to do this with a number of cross disciplinary, institutional and sectoral initiatives, and we look forward working with the inquiry and the future UK Committee on Research Integrity to ensure that efforts are coordinated, make efficient use of resources, and enable sharing of best practice.

Introduction

1. The UK Reproducibility Network (UKRN) is a peer-led consortium that aims to ensure the UK retains its place as a centre for world-leading research. It comprises three main elements: 1) Local Networks, which comprise informal, self-organising groups of researchers interested in issues of reproducibility, replicability and research quality, 2) Institutional Members, who have created senior academic positions focused on research improvement, and 3) External Stakeholder, which comprises funders, publishers, learned societies and other sectoral organisations. The activity across and between these elements, as well as the overall strategic direction of UKRN, is coordinated by a Steering Group.

2. The UKRN structure reflects the inter-connected nature of the research ecosystem. It also highlights the need for any initiatives intended to improve research quality to both be informed by the voice of the grassroots research community, as well as the needs of other stakeholders such as funders, and to be coordinated. It also represents an attempt to work collectively and collaboratively (rather than competitively). In our view, whilst a degree of competition is healthy, the extent to which academia has become hyper-competitive is problematic at both an individual and institutional level, and contributes to problems of low reproducibility and replicability.

3. The distinct elements of UKRN have each prepared submissions to this inquiry, where they address some of the questions posed – in particular the role of funders, institutions, researchers, publishers, and other organisations, and the potential impact of a UK Committee on Research Integrity. Here we synthesise common themes across these submissions, with a particular focus on the final question – how establishing a UK Committee on Research Integrity could impact the reproducibility “crisis” – and how organisations such as UKRN can support such a Committee.

An Interconnected System

4. In our view, the “crisis” narrative can be unhelpful, in part because it implies an acute situation that can be resolved, whereas in our view a framework for continuous improvement will be necessary and ultimately more effective. Whilst there are many reasons why specific research findings may not be robust, we agree that there is considerable room for improving the quality of scientific research outputs, and that this should generate improvements in the speed with which knowledge is generated, and in turn provide benefit to society.

5. As we have written previously: “Modern research-intensive universities present a paradox. On the one hand, they are dynamic, vibrant institutions where researchers use cutting-edge methods to advance knowledge. On the other, their traditions, structures, and ways of working remain rooted in the 19th century model of the independent scientist.” This underlying culture, where research groups operate effectively as artisanal small businesses, when combined with human cognitive biases and the current incentive structures, combine in our view to contribute to problems of poor reproducibility and replicability.

6. The research ecosystem is therefore complex and highly interconnected. Incentives, for example, may be embodied in institutional hiring and promotion practices, but are also influenced by the demands of research funders, the (explicit or implicit) requirements of journals and publishers, and the enthusiasm of researchers themselves to make discoveries that will advance their field. In the majority of cases, these influences are unconscious or implicit, and were introduced for the best of reasons but often have unintended consequences that affect reproducibility.

7. To bring about change, efforts will need to be coordinated across the research ecosystem. For example, open research practices (e.g., sharing data and code) will require supporting digital infrastructures, training in the skills necessary to make use of these infrastructures, mandates from funders and publishers to require open research practices where appropriate, monitoring of performance to create motivation for improvement (which in turn requires coordination across funders and publishers), and recognition of these practices in institutional hiring and promotion practices (again to create an incentive).

8. In other words, no one actor in the research ecosystem can bring about meaningful change alone. Indeed, individual actors may place themselves at a disadvantage, or create a burden that benefits nobody. For example, whilst there are advantages to individual researchers in engaging in open research practices, institutional recognition of this will be necessary for career advancement. And if this recognition is not sector-wide, then a researcher working in alignment with their own institutional framework may be at a relative disadvantage when moving to another institution if such practices are not valued there.

9. We therefore recommend that future work on reproducibility should focus on understanding how the research system affects reproducibility, and consider how a programme of measures across different sectors can work in concert to improve reproducibility. The UKRN developed a theory of change to improve research quality, which we would happily discuss with the inquiry, and the Committee once it is established.

The Role of a UK Committee on Research Integrity

10. In our view, the primary role of a UK Committee on Research Integrity should be to coordinate sectoral efforts to improve research quality across the research ecosystem, by focusing in particular on the research environment and the process of research. For example, open research practices are a possible means by which the quality of research can be improved, and if more granular contributions to the research process (i.e., the component parts of a research workflow, such as the study protocol, data, analysis code, etc.) are to be recognised, this will require (as noted above) coordination with researchers, institutions, funders, publishers, learned societies and a range of other sectoral organisations (including industry).

11. Critically, these efforts will require investment (e.g., in digital infrastructures and training to support open research practices), along with a more collaborative approach. Currently, local solutions (e.g., training, approaches to research assessment) are developed by individual institutions and organisations. This is inefficient and reduces interoperability. UKRN is producing statements for institutional use, for rapid adaptation and adoption. We are also developing train-the-trainer courses on open research practices that allow coordinated training to be delivered locally in a way that ensures a degree of common approach. We recommend continued and widening support for collaborative approaches to addressing reproducibility by funding initiatives that enable the sharing of resources and best practices across discipline and institutions.

12. In addition, this activity should be based on firm evidence for what works. There is an exciting level of innovation in the research ecosystem at present, including new publishing models and partnerships between funders and journals. However, innovations and approaches that prima facie appear likely to produce positive change may not deliver these in practice, or may have unintended consequences. We therefore recommend funding to support meta-research activity to evaluate the impact of new ways of working, to ensure that any benefits are understood and evidence, and any potential unintended consequences monitored. Such research should exploit the full range of tools available, including randomised controlled trials of effectiveness and could be coordinated across research funders to ensure efficient use of resources. We see a role for the UK Committee on Research Integrity in identifying where there is a need for such activity, and working with funders across the sector to ensure it is targeted the issues identified by the Committee.

Conclusion

13. Rather than viewing the current debate around the reproducibility and replicability of scientific findings as a “crisis”, it is more constructive in our view to view it as an opportunity to reflect on which aspects of scientific working practices continue to be effective; which can be improved; and which new ways of working can beneficially be introduced to the research ecosystem. We recommend taking a systems view of the factors affecting reproducibility, with a coordinated approach across disciplines, institutions and sectors. Doing this effectively will require, investment, and ongoing evaluation, allowing us to move to a model of continuous improvement at the level of the individual researcher, the institution, sectoral organisations, and the sector as a whole.

(September 2021)