Written Evidence Submitted by Imperial College London

(RRE0064)

● the breadth of the reproducibility crisis and what research areas it is most prevalent in;

More work is needed to provide robust data about the “reproducibility crisis”. However, we can indicate the anecdotal concerns below.

All science – not providing sufficient information/resources to allow experiments to be replicated

All science – insufficient data to support robust statistical analysis and/or address heterogeneity

Multidisciplinary science – researchers not always sufficiently familiar with pitfalls, controls across different disciplines

Discipline-specific issues

Psychology – e.g., failure to design appropriate experiments with robust controls

Life Sciences - e.g., standardization of biological constructs, cell lines, documentation of protocols, instrument performance

Clinical studies- e.g., difficult/impossible to share the full data due to confidentiality/data access issues; inadequate samples sizes, over-reliance on – and misapplication of – significance tests

Data intensive science – documentation, lack of curated data and analysis tools

Fields with strong vested interests inhibiting reproducibility and research integrity

As seen with research around health risks of smoking, climate science, …

Software - now a key element in undertaking/supporting research across almost all fields.

In a STEMB focused institution such as Imperial, software plays a significant role in supporting research and can also be a key source of challenges around reproducibility and research integrity. Thus, the breadth of such challenges is very wide. Unfortunately, there can be strong disincentives (e.g., significant additional time and resource requirements) for software developers to follow the good practices that help to mitigate these challenges – even though such good practices, e.g., use of version control systems for code, extensive testing and continuous integration for automated running of software builds and tests, often prove to be important in the long term.

Projects that involve complex (semi-)automated software-driven data processing or data generation pipelines are particularly prone to reproducibility challenges – which may persist even when even standard software development and management best practices have been applied.

We note that some valid science utilising imperfect information may not (yet) be reproducible, but it can still be useful to share progress to date – as happened during the COVID-19 pandemic. However, it is important that researchers and publishers present the uncertainties associated with research findings.

● the issues in academia that have led to the reproducibility crisis;

Drivers of funding and esteem

Early publication of speculative, exciting research can increase chances to win funding. Publication and grant funding may be harder to achieve if researchers are open about the weaknesses of their study. There is pressure to present all research as novel and positive – presenting it in the best possible light – including when trying to publish in high impact factor journals.

Not much incentive for busy researchers to ensure reproducibility – including robustness, maintainability and long-term sustainability of software – noting that reproducibility is often not specifically addressed in research proposals or final reports. In general, the burden of work to ensure reproducibility falls on the publishing researchers while others (e.g., the users of the research) benefit from this work.

Not much incentive (or funding) to reproduce the work of others or undertake meta research into the factors that improve reproducibility

Insufficient funding for infrastructure to share data (and other resources) at scale – e.g., to enable meta research

Insufficient funding/time/other resources for researchers, e.g., to document methodologies and protocols, to test and debug software, to pay independent experts to re-run statistical analyses before publication.

Limited access to major facilities that may be oversubscribed. Researchers are under pressure to publish data from time-limited experiments, often with little or no scope to re-run experiments. Furthermore, subsequent access may depend on publication of previous work.

Researchers sometimes over-promise in grant proposals in order to win funding and it is difficult to allocate time/resource to ensuring reproducibility of what has worked and been published, compared to further progressing the research project to achieve more outputs/funding.

Media

Media exposure can be important for academic career progression, winning funding and delivering impact but journalists can be motivated to make science stories as dramatic as possible, sometimes encouraging premature sharing of research findings and exaggeration of progress and/or impact.

Other drivers

Incentives: funding/recognition/career progression/publication stress novelty and being first (to publish, patent) with little penalty for later being proved wrong – and there is usually a low probability of being found to be wrong.

Pressure on academics with limited resources (particularly early career researchers) to frequently publish in high impact journals as a primary metric for their employment (promotions), success in funding applications, and even REF (geared to obtaining as many 4* papers which are judged so in part by impact factor).  Rigour has been explicitly included in the REF, but this does not specifically address reproducibility.

Too much competition rather than collaboration – leads to less sharing and openness - and therefore less discussion of best practice.  We recommend support for initiatives that enable more collaboration within and between disciplines and across institutions regarding reproducibility to ensure efficient use of resources, the sharing of best practice and a coordinated response across the research system

Work on reproducibility does not contribute to career performance metrics. We would recommend that incorporating consideration of reproducibility in career progression is looked at across institutions

Lack of training in “reproducibility good practice” – such training should be provided from undergraduate/postgraduate level, to ensure that as researchers gain experience, the principles and importance of reproducibility are already well understood.

• Software development is often not taught as a core skill at UG level – and not formally taught at PG level. Where software development training is provided, this often doesn’t cover best practices and software project management techniques that are key aspects for supporting reproducibility of research outputs. Knowing how to write code and make it available to others doesn’t equate to reproducible outputs. Self-taught research software developers can be unfamiliar with testing techniques or approaches for running automated testing which help to identify whether outputs change, or errors are introduced. Such testing can form a significant percentage of development time and, if code has no inherent academic value beyond its use for a specific publication, there is a disincentive to provide this.
• The impact on reproducibility of software outputs can be magnified if the software undergoes subsequent further development by other developers. No modern software is the work of just one individual - even where one person has written some code, it will almost always rely on “libraries” of code written by others. Since these libraries typically change over time, their behaviour/inputs/outputs may also change, and careful planning is needed to ensure that an equivalent environment can be recreated to re-generate research results
• Both the developers and users of research software have responsibility for ensuring that research outputs using software are reproducible. Developers should provide the resources required to enable others to reproduce their results using their software and should provide a record of any subsequent changes for future users. Users of software should test that it functions as intended before applying it to new research.

● the role of the following in addressing the reproducibility crisis:

- research funders, including public funding bodies should

Fund training

Fund infrastructure, e.g., for sharing data, other resources

Provide funding for a for discussion of reproducibility and sharing of best practice (e.g., https://www.ukrn.org/)

Fund meta research

Use funding to drive collaboration, dialogue, sharing and best-practice. For example, consortia funding that enables groups to pool samples and resources, increasing sample sizes, multiple methodologies and standardize data collection and analysis

Require the approach to reproducibility to be addressed in all research proposals – ideally with a uniform approach across funders

Fund the additional work required to enable reproducibility. Assess whether investigators are aware of issues and whether sufficient resource has been allocated in research proposals

Check compliance with reproducibility good practice after projects have ended

- research institutions and groups should

Enable cross-institutional (Faculty/Department/research group) dialogue to instil reproducibility in research culture, noting discipline-specific differences

Develop institutional policies to enable/ensure reproducibility

Give more recognition to non-publication research outputs (software, methodologies, protocols, data sets, biological constructs, …)

Provide training – staff (raising awareness and providing guidance) and students (Graduate Schools)

Consider designating reproducibility experts/advisors with specific domain expertise

Allocate time and resources to address reproducibility issues

Provide adequate infrastructure to enable sharing of data, software, protocols, constructs, etc

Support and monitor compliance – of actual practice rather than check boxes on proposals

Highlight good practice, e.g., TU Delft Research Software Policy (https://doi.org/10.5281/zenodo.4629635) and Research Data Framework Policy (https://doi.org/10.5281/zenodo.2573159)

- individual researchers should

Use best practice checklists (if available) before publication – sharing methodologies, protocols, data, constructs, etc., with publications
(can be discipline-specific)

Review regularly, e.g., at PRDP

Recognise the potential need to develop new technical skills around reproducibility

Recognise the need to allocate time and resources to address reproducibility issues

For software, understand that a task is not just about writing code but about an ecosystem of associated tools and practices that ensure that software works as expected and continues to do so into the future.

Promote best practice in their communities (journals, conferences, reviewing, …)

- publishers should

Appropriately value underpinning research outputs (data, software, methodologies, protocols, constructs, etc.) and require evidence of reproducibility being addressed for all publications - analogous to how Journal of Open Source Software (JOSS), Journal of Open Research Software (JORS) and SoftwareX aim to drive good practice with research software.

Encourage authors to share weaknesses of their studies as well as strengths – make this good practice and not a reason to reject manuscript.

Require authors to register/share (interpretable) data, methodologies, protocols, software, as appropriate– see https://www.imperial.ac.uk/computational-methods/software-data/; https://www.protocols.io/; https://github.com; https://zenodo.org/; etc

Consider pre-registration of methodologies/protocols, before a study has begun – see https://osf.io/8v2n7/ – or formal registration of research plans (see https://conbio.onlinelibrary.wiley.com/doi/full/10.1111/cobi.13342) for discussion in context of ecology and conservation

Consider supporting a reproducibility badging scheme where software can be pre-assessed for reproducibility/sustainability and given a certification.

Checklists to remind peer-reviewers to assess reproducibility and other issues of research integrity (e.g., see https://www.nature.com/articles/s41559-018-0545-z)

- Governments and the need for a unilateral response / action.

Governments should establish a standard of reproducibility that is universally accepted as good practice and becomes the norm, e.g., for funders and for publications. This could be explicitly highlighted in the assessment criteria for REF.

This could include developing a set of national guidelines that can support and advise individual researchers directly and act as a basis upon which institutions could develop their own guidance.

● what policies or schemes could have a positive impact on academia’s approach to reproducible research;

Encourage universities/institutions to discuss and share best practice

Encourage universities to develop institutional policies to address reproducibility and support/monitor their compliance

Ensure that funders award support for investigators to address reproducibility

Encourage funders to support studies to reproduce important results and to support meta research

Encourage funders to support institutions/agencies to curate non-publication outputs (data, methodologies, protocols, software) to support reproducibility. This could include central government/research council funded subscriptions to core services that support best practices (e.g., version control, continuous integration/testing, software archiving, etc).

Encourage funders to support institutions/agencies that can monitor/test/certify reproducibility, including as a service to authors wishing to publish their outputs.

Encourage funders to require and support training to upskill the research workforce in gaining the necessary knowledge and skills to develop reproducible research outputs, including

effective version control / versioning of software to know which specific version of a code was used to produce specific results to ensure they are reproducible

use of unique identifiers (e.g., DOIs) to identify/reference data and code - services to archive data and code and provide these identifiers are becoming more common.

Encourage funders and publishers to work together to develop strategies and policies to address reproducibility such as pre-registration and registered reports, checklists

Reduce use of journal impact factors, including during evaluation of research staff and institutions. Some high impact journals have some of the highest retraction rates per journal (https://towardsdatascience.com/on-retractions-in-biomedical-literature-1565e773559e) and it has been reported that the quality of a study does not improve with higher impact factor of the journal where they are published (https://www.nature.com/articles/d41586-021-02486-7#ref-CR7).

Investigate improving the rigour assessments of the REF to make assessment of reproducibility more explicit

Consider introducing ‘reproducibility factor’ for journals to encourage practices to facilitate reproducibility. A ‘TOP’ factor has been proposed for those journals supporting transparency and openness promotion (https://osf.io/9f6gx/wiki/Guidelines/?_ga=2.120857167.636993759.1584446687-856647102.1584446687)

● how establishing a national committee on research integrity under UKRI could impact the reproducibility crisis.

A national committee could work across disciplines and sectors (with funders, publishers, institutions, individuals at different career stages, …) to provide unambiguous guidance and resources with respect to good (expected) practice and sharing research outputs (data, software, protocols, constructs, etc.), and to co-ordinate progress and monitoring of compliance.

(September 2021)