Written Evidence Submitted by Imperial College London
● the breadth of the reproducibility crisis and what research areas it is most prevalent in;
More work is needed to provide robust data about the “reproducibility crisis”. However, we can indicate the anecdotal concerns below.
All science – not providing sufficient information/resources to allow experiments to be replicated
All science – insufficient data to support robust statistical analysis and/or address heterogeneity
Multidisciplinary science – researchers not always sufficiently familiar with pitfalls, controls across different disciplines
Psychology – e.g., failure to design appropriate experiments with robust controls
Life Sciences - e.g., standardization of biological constructs, cell lines, documentation of protocols, instrument performance
Clinical studies- e.g., difficult/impossible to share the full data due to confidentiality/data access issues; inadequate samples sizes, over-reliance on – and misapplication of – significance tests
Data intensive science – documentation, lack of curated data and analysis tools
Fields with strong vested interests inhibiting reproducibility and research integrity
As seen with research around health risks of smoking, climate science, …
Software - now a key element in undertaking/supporting research across almost all fields.
In a STEMB focused institution such as Imperial, software plays a significant role in supporting research and can also be a key source of challenges around reproducibility and research integrity. Thus, the breadth of such challenges is very wide. Unfortunately, there can be strong disincentives (e.g., significant additional time and resource requirements) for software developers to follow the good practices that help to mitigate these challenges – even though such good practices, e.g., use of version control systems for code, extensive testing and continuous integration for automated running of software builds and tests, often prove to be important in the long term.
Projects that involve complex (semi-)automated software-driven data processing or data generation pipelines are particularly prone to reproducibility challenges – which may persist even when even standard software development and management best practices have been applied.
We note that some valid science utilising imperfect information may not (yet) be reproducible, but it can still be useful to share progress to date – as happened during the COVID-19 pandemic. However, it is important that researchers and publishers present the uncertainties associated with research findings.
● the issues in academia that have led to the reproducibility crisis;
Drivers of funding and esteem
Early publication of speculative, exciting research can increase chances to win funding. Publication and grant funding may be harder to achieve if researchers are open about the weaknesses of their study. There is pressure to present all research as novel and positive – presenting it in the best possible light – including when trying to publish in high impact factor journals.
Not much incentive for busy researchers to ensure reproducibility – including robustness, maintainability and long-term sustainability of software – noting that reproducibility is often not specifically addressed in research proposals or final reports. In general, the burden of work to ensure reproducibility falls on the publishing researchers while others (e.g., the users of the research) benefit from this work.
Not much incentive (or funding) to reproduce the work of others or undertake meta research into the factors that improve reproducibility
Insufficient funding for infrastructure to share data (and other resources) at scale – e.g., to enable meta research
Insufficient funding/time/other resources for researchers, e.g., to document methodologies and protocols, to test and debug software, to pay independent experts to re-run statistical analyses before publication.
Limited access to major facilities that may be oversubscribed. Researchers are under pressure to publish data from time-limited experiments, often with little or no scope to re-run experiments. Furthermore, subsequent access may depend on publication of previous work.
Researchers sometimes over-promise in grant proposals in order to win funding and it is difficult to allocate time/resource to ensuring reproducibility of what has worked and been published, compared to further progressing the research project to achieve more outputs/funding.
Media exposure can be important for academic career progression, winning funding and delivering impact but journalists can be motivated to make science stories as dramatic as possible, sometimes encouraging premature sharing of research findings and exaggeration of progress and/or impact.
Incentives: funding/recognition/career progression/publication stress novelty and being first (to publish, patent) with little penalty for later being proved wrong – and there is usually a low probability of being found to be wrong.
Pressure on academics with limited resources (particularly early career researchers) to frequently publish in high impact journals as a primary metric for their employment (promotions), success in funding applications, and even REF (geared to obtaining as many 4* papers which are judged so in part by impact factor). Rigour has been explicitly included in the REF, but this does not specifically address reproducibility.
Too much competition rather than collaboration – leads to less sharing and openness - and therefore less discussion of best practice. We recommend support for initiatives that enable more collaboration within and between disciplines and across institutions regarding reproducibility to ensure efficient use of resources, the sharing of best practice and a coordinated response across the research system
Work on reproducibility does not contribute to career performance metrics. We would recommend that incorporating consideration of reproducibility in career progression is looked at across institutions
Lack of training in “reproducibility good practice” – such training should be provided from undergraduate/postgraduate level, to ensure that as researchers gain experience, the principles and importance of reproducibility are already well understood.
● the role of the following in addressing the reproducibility crisis:
- research funders, including public funding bodies should
Fund infrastructure, e.g., for sharing data, other resources
Provide funding for a for discussion of reproducibility and sharing of best practice (e.g., https://www.ukrn.org/)
Fund meta research
Use funding to drive collaboration, dialogue, sharing and best-practice. For example, consortia funding that enables groups to pool samples and resources, increasing sample sizes, multiple methodologies and standardize data collection and analysis
Require the approach to reproducibility to be addressed in all research proposals – ideally with a uniform approach across funders
Fund the additional work required to enable reproducibility. Assess whether investigators are aware of issues and whether sufficient resource has been allocated in research proposals
Check compliance with reproducibility good practice after projects have ended
- research institutions and groups should
Enable cross-institutional (Faculty/Department/research group) dialogue to instil reproducibility in research culture, noting discipline-specific differences
Develop institutional policies to enable/ensure reproducibility
Give more recognition to non-publication research outputs (software, methodologies, protocols, data sets, biological constructs, …)
Provide training – staff (raising awareness and providing guidance) and students (Graduate Schools)
Consider designating reproducibility experts/advisors with specific domain expertise
Allocate time and resources to address reproducibility issues
Provide adequate infrastructure to enable sharing of data, software, protocols, constructs, etc
Support and monitor compliance – of actual practice rather than check boxes on proposals
Highlight good practice, e.g., TU Delft Research Software Policy (https://doi.org/10.5281/zenodo.4629635) and Research Data Framework Policy (https://doi.org/10.5281/zenodo.2573159)
- individual researchers should
Use best practice checklists (if available) before publication – sharing methodologies, protocols, data, constructs, etc., with publications
(can be discipline-specific)
Review regularly, e.g., at PRDP
Recognise the potential need to develop new technical skills around reproducibility
Recognise the need to allocate time and resources to address reproducibility issues
For software, understand that a task is not just about writing code but about an ecosystem of associated tools and practices that ensure that software works as expected and continues to do so into the future.
Promote best practice in their communities (journals, conferences, reviewing, …)
- publishers should
Appropriately value underpinning research outputs (data, software, methodologies, protocols, constructs, etc.) and require evidence of reproducibility being addressed for all publications - analogous to how Journal of Open Source Software (JOSS), Journal of Open Research Software (JORS) and SoftwareX aim to drive good practice with research software.
Encourage authors to share weaknesses of their studies as well as strengths – make this good practice and not a reason to reject manuscript.
Require authors to register/share (interpretable) data, methodologies, protocols, software, as appropriate– see https://www.imperial.ac.uk/computational-methods/software-data/; https://www.protocols.io/; https://github.com; https://zenodo.org/; etc
Consider pre-registration of methodologies/protocols, before a study has begun – see https://osf.io/8v2n7/ – or formal registration of research plans (see https://conbio.onlinelibrary.wiley.com/doi/full/10.1111/cobi.13342) for discussion in context of ecology and conservation
Consider supporting a reproducibility badging scheme where software can be pre-assessed for reproducibility/sustainability and given a certification.
Checklists to remind peer-reviewers to assess reproducibility and other issues of research integrity (e.g., see https://www.nature.com/articles/s41559-018-0545-z)
- Governments and the need for a unilateral response / action.
Governments should establish a standard of reproducibility that is universally accepted as good practice and becomes the norm, e.g., for funders and for publications. This could be explicitly highlighted in the assessment criteria for REF.
This could include developing a set of national guidelines that can support and advise individual researchers directly and act as a basis upon which institutions could develop their own guidance.
● what policies or schemes could have a positive impact on academia’s approach to reproducible research;
Encourage universities/institutions to discuss and share best practice
Encourage universities to develop institutional policies to address reproducibility and support/monitor their compliance
Ensure that funders award support for investigators to address reproducibility
Encourage funders to support studies to reproduce important results and to support meta research
Encourage funders to support institutions/agencies to curate non-publication outputs (data, methodologies, protocols, software) to support reproducibility. This could include central government/research council funded subscriptions to core services that support best practices (e.g., version control, continuous integration/testing, software archiving, etc).
Encourage funders to support institutions/agencies that can monitor/test/certify reproducibility, including as a service to authors wishing to publish their outputs.
Encourage funders to require and support training to upskill the research workforce in gaining the necessary knowledge and skills to develop reproducible research outputs, including
effective version control / versioning of software to know which specific version of a code was used to produce specific results to ensure they are reproducible
use of unique identifiers (e.g., DOIs) to identify/reference data and code - services to archive data and code and provide these identifiers are becoming more common.
Encourage funders and publishers to work together to develop strategies and policies to address reproducibility such as pre-registration and registered reports, checklists
Reduce use of journal impact factors, including during evaluation of research staff and institutions. Some high impact journals have some of the highest retraction rates per journal (https://towardsdatascience.com/on-retractions-in-biomedical-literature-1565e773559e) and it has been reported that the quality of a study does not improve with higher impact factor of the journal where they are published (https://www.nature.com/articles/d41586-021-02486-7#ref-CR7).
Investigate improving the rigour assessments of the REF to make assessment of reproducibility more explicit
Consider introducing ‘reproducibility factor’ for journals to encourage practices to facilitate reproducibility. A ‘TOP’ factor has been proposed for those journals supporting transparency and openness promotion (https://osf.io/9f6gx/wiki/Guidelines/?_ga=2.120857167.636993759.1584446687-856647102.1584446687)
● how establishing a national committee on research integrity under UKRI could impact the reproducibility crisis.
A national committee could work across disciplines and sectors (with funders, publishers, institutions, individuals at different career stages, …) to provide unambiguous guidance and resources with respect to good (expected) practice and sharing research outputs (data, software, protocols, constructs, etc.), and to co-ordinate progress and monitoring of compliance.