Written Evidence Submitted by the UK Reproducibility Network Local Network Leads
Dr Suzanne L. K. Stewart, University of Chester, UK
Dr Charlotte R. Pennington, Aston University, Birmingham, UK
Gonçalo R. da Silva, Queen’s University Belfast, UK
Nick Ballou, Queen Mary University of London, UK
Dr Jessica Butler, University of Aberdeen, UK
Prof. Zoltan Dienes, University of Sussex, UK
Prof. Caroline Jay, University of Manchester, UK
Dr Stephanie Rossit, University of East Anglia, UK
Dr Anna Samara, University of Greenwich, UK
On behalf of the UKRN Local Network Leads.
UKRN Local Networks are grassroots networks of researchers at different institutions. These are self-organising, but are encouraged to ensure their membership is open to academic and research staff (at all levels) and professional services staff. The UKRN currently includes Local Networks across 57 institutions.
1. The UK Reproducibility Network (UKRN) is a peer-led consortium that aims to ensure the UK retains its place as a centre for world-leading research. Within its wider advocacy and training work to improve research integrity and reproducibility, UKRN seeks to link (1) local networks of researchers, (2) institutional leads representing universities and research institutes that have formally joined UKRN, and (3) stakeholder organisations such as funders, publishers, and policymakers. We represent the voice of the UKRN’s Local Network Leads (LNLs), as well as the voices of the numerous individual researchers from diverse academic disciplines and research areas, across all career stages, and throughout the UK from 57 local networks that we convene.
2. In recent years there has been increasing concern regarding the reproducibility and replicability of research (e.g., Dienlin et al., 2020; Niven et al., 2018; Open Science Collaboration, 2015). Both are fundamental to the research process and underpin the credibility, applicability, and impact of research findings in society. We applaud this inquiry’s mission to address the reproducibility crisis. However, the word “crisis” implies passivity. Rather, in response to second topic of the inquiry’s call for evidence, the issues that have led to the reproducibility crisis, the current state of affairs -- in which much research has only the appearance of quality and which crumbles at the barest scrutiny -- was engendered by mutual reinforcement of cultural incentives for employment, promotion, prestige, and profit that prioritised fast, uncomplicated, attention-grabbing, and ambitious research. As in all cycles of boom and bust, the research ecosystem is now revealed to be without a strong foundation (for reviews see Munafo et al., 2017; Vitae, 2020).
3. It is apparent that all of us in the research ecosystem must seize this opportunity for coordinated change. But to institute effective reforms, we must identify our common goals and agree on a shared path towards them. We argue that:
4. The difficult question for all of us in the research ecosystem is, “How do we coordinate our efforts to build a foundation of open and transparent research practice that supports research quality?” In the action-centred evidence we set out below, we attempt to answer this question via the third topic of the inquiry’s call for evidence, specifically “the role of research funders, including public funding bodies; research institutions and groups; individual researchers; publishers; and Governments in addressing the reproducibility crisis, and the need for a unilateral response/action.”
Coordinating a Positive Culture of Open and Transparent Research Practice
5. Problems of research integrity are grounded in a lack of awareness and habitual use of poor research practices, rather than an intention to mislead. Thus, a national Government committee on research integrity should focus on positive actions and incentives that support both local and national progression towards open and transparent research practice.
6. Such a committee must coordinate with Institutions to improve working cultures. Relentless pressure to publish research outputs and acquire grant funding is commonplace, as is the resulting negative impact on staff wellbeing (see Wellcome report). Such pressure is counter-productive; individuals thrive when they feel supported, recognised for effort rather than achievements, and trusted with autonomy. This pressure incentivises closed and opaque research practice and shortcuts that, in the short-term, lead to a high quantity of research outputs, but which, in the long-term, can harm research quality.
7. Instead, Institutions should reward and encourage open and transparent research practice through their incentive structures for Individuals, for example by including consideration of open and transparent practice into their hiring, induction, probation, promotion, workload, and professional development policies and frameworks (e.g., using the Résumé for Researchers format). Government can, in turn, incentivise Institutions to execute such reforms by requiring evidence and data that such policies are adopted and implemented. Any such policy changes also need clear coordination with Equality, Diversity, and Inclusion policies and reforms.
8. Such coordinated efforts should be guided by national policies that signal the value of open and transparent research practice. Government should ensure that its commitment to open and transparent research in its Research & Development Roadmap is fully executed. This can be underpinned by a specific national policy that incentivises positive, concrete actions for All across the research ecosystem, such as the actions that we and others set out as evidence for this inquiry. This policy should borrow examples of effective practice from other nations. For example, both France (Ministère de L’Enseignement Supérieur, de la Recherche et de L’Innovation, 2021) and the Netherlands (VSNU et al., 2020) have produced systematic, detailed, concrete plans for progressing open and transparent research practice.
9. Indeed, Government can set incentives for Individuals, Institutions, Funders, and Publishers to increase engagement with open and transparent research practice through its own research mechanisms: its own funding arms, its own institutions that conduct research and have their own ethics and governance processes (e.g., the NHS, the Armed Forces), and through its Higher Education policies. Such cultural change will be fostered more successfully through the mechanism of mutually reinforced, coordinated incentives rather than strict mandates.
10. In a similar way, Funders can help incentivise Individuals and Institutions to engage with open and transparent research practice by strategically prioritising regular calls for projects on meta-research (how researchers do research), on open and transparent research practices in relevant discipline(s), and for replications of published research studies. This would serve four key purposes: (a) demonstrating to the research community that such areas are valued and important to the research ecosystem, (b) helping the research community acquire new data about effective improvements to research practice to support evidence-based actions in the future, (c) encouraging individuals and institutions to adopt open and transparent practices and replication work, and (d) shifting the incentive structures to reward open and transparent research practices and meta-research.
11. Government has already recognised through UKRI that open access research outputs are valuable. This positive cultural change should be followed now by a coordinated, across-the-board effort by Publishers to support open access policies and publishing platforms while actively discouraging the view that open access-only publishing options (e.g., open access journals) are somehow less prestigious or rigorous. Funders, Institutions (through subscriptions), and Individuals (through targeted research output outlets) can incentivise Publishers to broaden and improve open access policies, as well as other avenues that elevate open and transparent research practice, such as pre-registration, Registered Reports, and mandates for the sharing of data, research materials, and other types of output (e.g., software).
Coordinating a Unified Stance for Open and Transparent Research Practice
12. Institutions should coordinate and adopt common policies, guidance, and training for monitoring and improving efforts for reproducibility, openness, and quality. For example, UKRN Institutional Leads have worked together with Local Network Leads to produce a series of common statements for use by the sector on topics such as transparency in research. Other sectoral organisations (e.g., Universities UK) can also contribute. This collective and collaborative sectoral approach can help improve research integrity and reproducibility by encouraging a common approach to these issues, informed by the voice of grassroots researchers.
13. Institutions, Government, and others (such as Industry) should coordinate the inclusion of open and transparent research practice into their research ethics and governance approval processes. Such inclusion should take a flexible approach which recognises that open and transparent practice varies by research area and, therefore, input from Individuals is necessary to ensure that processes are executable and comply with other relevant frameworks (e.g., Funder mandates, and legal frameworks such as GDPR).
Coordinating the Foundations for Open and Transparent Research Practice
14. Institutions, Funders, and Publishers should seek to continuously improve infrastructure (including digital infrastructure) that supports research. This could include coordinated, cross-sector development and/or maintenance of (for example) databases, digital storage, servers, software, repositories, and various researcher-led initiatives. There should be collaboration with Individuals and coordinating organisations such as UKRN to ensure infrastructure is fit-for-purpose, avoids duplication, and is interoperable.
15. All members of the research ecosystem must recognise that a strong foundation for open and transparent research practice includes knowledge. Thus, Institutions should recognise the diverse range of specialists who contribute to the openness and transparency of research projects, beyond academic researchers. This includes (but is not limited to) data managers, research software engineers, data scientists, statisticians, laboratory managers, technicians, and compliance officers. To ensure a sustained and integrated commitment to improving open and transparent research practice, Institutions should ensure core-funded positions for these roles, with appropriate routes for career progression and promotion. Funders should support such roles in their schemes, and Publishers should promote creditorship (in addition to authorship) to recognise the contributions of these key individuals.
16. Foundational knowledge is relevant for individual researchers as well as individuals who support the research ecosystem in other ways, such as those who work for Publishers, Funders, and Government. Thus, all sectors of the research ecosystem should coordinate and mutually reinforce openly accessible training and guides on open and transparent research practice. For example, Publishers should facilitate continuous professional development training and infrastructure related to the publishing of outputs, including data management, licencing, and digital object identifiers, which would be of benefit to researchers, editors, and reviewers. Similarly, Funders should provide detailed guidance and training on the open and transparent practices that they require and/or encourage. This guidance and training should be systematically reviewed and updated as required to reflect ongoing developments.
17. Of course, Individual researchers should be supported and incentivised by all others in the research ecosystem to engage in continuous professional development, with an emphasis on open and transparent research as appropriate for their research area. Such development should include a focus on the digital skills and infrastructure required to engage in open and transparent research practice (e.g., sharing data and code). Individuals must recognise their responsibility to ensure their knowledge and skills remain current. This can only occur, however, if cultural changes at Institutions include the promotion of, and the necessary time for, continuous professional development of research skills for Individuals, at all career stages, in addition to employing specialists (see point 15).
18. In turn, Individuals have a responsibility to integrate open and transparent research practice into their teaching and training of undergraduate and postgraduate students as well as junior researchers whom they line manage. Institutions share this responsibility and can support the longevity of open and transparent research practice by coordinating training, positive incentive and reward structures, infrastructure, and policies relating to workload and promotion.
Coordinating the routinisation of open and transparent research practice
19. A coordinated effort must be made to ensure that open and transparent research practice becomes the norm. All members of the research ecosystem have particular actions they can lead, while reinforcing the others, to integrate openness and transparency into the everyday practice of research. The points below exemplify these actions.
20. Individuals must integrate open and reproducible practices from the beginning of the research process, including in all relevant ethics and governance processes and in applying for and securing research funding. Individuals should be encouraged to use research infrastructure (e.g., software) that supports open and transparent practices throughout the research workflow, as well as publishing routes (e.g., Registered Reports) that can be integrated from the beginning of the research process. Consideration of open and transparent practices only at the end of a research project is too late, and risks becoming a box-ticking exercise only.
21. Funders should require, wherever possible, a plan for open and transparent research practices in funding applications, as appropriate to the discipline. Applicants should demonstrate whether and how they will share (for example) research data, original materials and protocols, software and code, research workflows, and pre- and post-publication outputs. Depending on career stage, applicants can also be asked to demonstrate a previous track record of verifiable open and transparent research practices. Funders can ask for these plans and track records to be reviewed and evaluated as part of the decision-making process for funding applications.
22. Funders should require confirmation that open and transparent research practices have been followed in funded projects (e.g., in final reports or via Researchfish). Tracking the locations of shared data and other intermediate outputs would allow funders to build a database of available products/outputs and resources that can efficiently support future work and the development of research tools and infrastructure.
23. Publishers should publish rigorous replication studies, as well as outputs on the process of research that can help Individuals, Institutions, and others improve their own work (e.g., tutorials on using digital platforms). As other actions to improve the adoption of open and transparent research practice take hold, interest in such outputs would continue to increase, and for-profit publishers would be implicitly incentivised by such changes in supply-and-demand.
24. Publishers should develop rigorously applied editorial policies that centre openness and transparency. One area is systematically checking for compliance with practices that support openness and transparency. For example, if data are required to be shared, publishers should mandate checks that ensure the data are openly accessible both before and after publication (see American Journal of Political Science). Such checks can include specific statistical and analysis code reviews as appropriate. While many publishers have such policies, they are not routinely enforced. Another area is systematically basing the review and selection of outputs on methodological rigour, openness, and transparency (the hallmarks of quality) rather than novelty or the nature of findings.
25. Publishers should have digital word-limit-free submission formats to encourage full and detailed disclosure of methodology, analytical decisions, and pilot work. Word limits that were last relevant in the days of print-only publishing force researchers to omit information that is essential for reproducibility and transparency, meaning that research quality suffers unnecessarily.
26. Government, Publishers, Funders, Institutions, and Individuals should recognise the value of distributed laboratory networks and collaborative team science in relevant disciplines as models of open and transparent research practice. Such large-scale collaborations have huge and untapped potential for producing impactful, reproducible, and reliable research findings, and for effectively pooling resources to minimise research waste. Government should support and incentivise such work through funding organisations.
27. As active researchers, we recognise both the challenges – but also the far-reaching opportunities – associated with committing to more open and transparent research practice as part of a suite of broader cultural changes. The burden of such changes must not rest solely or even primarily on individual researchers.
28. Researchers’ behaviours are a response to the structure and incentives of the ecosystem in which they work; thus, it is imperative that organisational stakeholders such as Institutions, Funders, Publishers, and Government work with and for Individuals.
29. To do so, all members of the research ecosystem must progress concrete actions, such as those that we have suggested here, or risk being left in a cycle of discussion where nothing changes and research quality stagnates or deteriorates. This approach should be collaborative, taking advantage of the inter-connected nature of the UK higher education ecosystem, and the existence of coordinating organisations such as UKRN.
AJPS Verification Policy. (n.d.). American Journal of Political Science. Retrieved September
24, 2021, from https://ajps.org/ajps-verification-policy/
Dienlin, T., Johannes, N., Bowman, N. D., Masur, P. K., Engesser, S., Kümpel, A. S., Lukito,
J., Bier, L. M., Zhang, R., Johnson, B. K., Huskey, R., Schneider, F. M., Breuer, J., Parry, D. A., Vermeulen, I., Fisher, J. T., Banks, J., Weber, R., Ellis, D. A.,...de Vreese, C. (2020). An agenda for open science in communication. Journal of Communication, 71, 1-26. https://www.doi.org/10.1093/joc/jqz052.
Metcalfe, J., Wheat, K., Munafò, M., & Parry, J. (2020). Research integrity: A landscape
Ministère de L’Enseignement Supérieur, de la Recherche et de L’Innovation. (2021).
Second national plan for open science. https://www.ouvrirlascience.fr/second-national-plan-for-open-science/
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du
Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, 0021. https://doi.org/10.1038/s41562-016-0021
Niven, D. J., McCormick, T. J., Straus, S. E., Hemmelgarn, B. R., Jeffs, L., Barnes, T. R. M.,
& Stelfox, H. T. (2018). Reproducibility of clinical research in critical care: A scoping review. BMC Medicine, 16, 26. https://www.doi.org/10.1186/s12916-018-1018-6.
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science.
Science, 349, 6251. https://www.doi.org/10.1126/science.aac4716.
Researchfish. Retrieved September 24, 2021, from https://researchfish.com/
UK Reproducibility Network. (n.d.). Common statements. https://www.ukrn.org/common-statements/
VSNU, KNAW, & NWO. (2020). Strategy Evaluation Protocol.
Wellcome Trust. (2020). What researchers think about the culture they work in.