Written Evidence Submitted by King’s College London
(RRE0091)
1 Introduction
1.1. King’s College London is a world-leading, research-intensive institution. Our core vision for making the world a better place is driven by our research endeavours. We deliver research to inform and innovate by generating knowledge, creating impact, and addressing global challenges.
1.2. The KCL Research Integrity Office promotes good conduct in research, supporting the university’s research community through the provision of training and guidance and the development of institutional policies and procedures. We are committed to ensuring that the research conducted by our staff and students is consistently of the highest quality and conforms to the most rigorous standards. The Office is located within the Department of Research Governance, Ethics & Integrity within the Research Management & Innovation Directorate and we work closely with other research support staff, with researchers and with central functions such as Libraries and Collections and Human Resources to foster a positive and supportive research culture.
1.3. As a research-intensive institution and employer of researchers we recognise our responsibility to engage in national conversations concerning research integrity, policy, and strategy. We are committed to external engagement, liaising, and collaborating with colleagues across the sector as an institutional member of the UK Reproducibility Network (UKRN), member of the Russell Group Research Integrity Forum, founding member of the London Research Integrity Consortium and a subscriber member of UK Research Integrity Office. Reflective of this commitment, we welcome the opportunity to submit evidence to this inquiry.
1.4. The KCL Research Integrity Office has prepared this response in collaboration with the College’s Research Integrity Champions and Advisors[1] with contributions from our UKRN Local Network Lead.[2] Research Integrity Champions are positions typically held by the Vice Deans for Research from each of our nine Faculties and are senior advocates for good conduct in research. Research Integrity Advisors are individuals from all career stages that have been nominated for their pursuit of excellence and conduct of research to the highest standards, or the promotion of the same, and act as local points of contact for queries on research integrity or misconduct.
2.1. The prevalence of irreproducible research findings poses significant challenges for funders, research organisations and governments. We believe all stakeholders have a responsibility to use sober and accurate language to discuss these challenges and question whether “crisis” risks overgeneralising a complex set of issues and may allow for misrepresentation in the media which has the potential to needlessly damage public trust. Referring to irreproducible research findings as a “crisis” may also imply that this is an acute issue that can be swiftly resolved, rather than a characteristic embedded in our current research culture.
2.2. The breadth of reproducibility issues appears wide, but lack of standardised metrics or definitions means this is unfeasible to quantify or benchmark at an institutional or discipline level. Biomedical and STEM subjects across clinical, translational, and basic fields are understood to be most affected but the impact on the social sciences is being increasingly recognised.
2.3. For many subjects, outside of STEM, particularly within Arts & Humanities, the concept of reproducibility is not applicable to the research methods employed and terms such as transparency are more appropriate to discuss the rigour of the research findings. If we approach reproducibility through broader definitions, we can engage with a broader range of researchers to better understand the scale of the issue in wider disciplinary areas.
3.1. The issues in academia that have led to irreproducible research findings are complex, multifaceted, and not limited to the suggestions below. They may be historical, systemic, and/or cultural and will require a co-ordinated and iterative effort by a range of stakeholders across the sector to address them.
3.2. A competitive and high-pressured environment places undue pressure on researchers in all disciplines. Time-limited research funding, high workloads, and the pressure to publish means researchers are compelled to produce and publish results in a short period of time regardless of the confidence in the results. This environment is underpinned by flawed systems of research and researcher evaluation; an over-reliance on publication records, narrow hiring and promotion criteria and the use of metrics such as the h-index contribute to the pressure to publish and value quantity over the quality of research. These factors can incentivise poor research practices, such as incomplete validation, over-interpretation, and misapplied statistical analysis. For early career researchers this is compounded by a reliance on fixed-term contracts, often tied to external funding where high-impact publications are necessary to compete for a limited number of permanent academic positions.
3.3. Where research endeavours produce null, negative, replicatory or confirmatory results there is little incentive or opportunity for researchers to share or publish the findings. There is a reasonable perception amongst researchers that publications presenting null results will receive a low number of citations and will not benefit career progression. There is also a perception that null results are also more difficult to publish due to lack of novelty which results in a higher rate of rejection from journals. Despite recent efforts from some publishers to provide platforms for these results, a cultural shift is required to value null or negative results as highly as novel or high-impact outputs. This issue is both reflected in the types of research funding available, which tends to value novelty or impact, and exacerbated by the publishing industry in which positive results are selectively overrepresented and this has been extensively reported on.
3.4. A degree of irreproducibility can be attributed to flawed experimental design, methodology or analysis and this can be associated with a lack of skills and training but is inseparable from the high-pressure environment and the lack of time or resources. To some extent, some irreproducibility arising from experimental flaws will be unavoidable given the degree of uncertainty attributed to knowledge generation where sources of variability are unknown or hard to control for. Assessing and defining these contextual variables is a vital part of the progress of knowledge generation.
4.1. Considering the context of irreproducibility issues within our research culture, many of the strategies suggested below are likely to have a positive impact on the integrity of research more broadly and are not necessarily limited to improving reproducibility.
4.2. Role of research funders in addressing reproducibility “crisis”
4.2.1. Research funders must ensure adequate availability and structures of funding that incentivise rigorous, high-quality research studies. In many cases this may require additional time, extension of grants, and additional resources such as platforms for data sharing or funding for bigger sample sizes, longer-term experiments or for replication or validatory studies. In addition to individual funding schemes, consideration should be paid to how short-term and non-fully economically costed funding awards impact researcher contract lengths and the ability of research institutions to provide stability and support for researcher’s careers. This relationship between funding, research careers and the impact on research quality and culture was explored in the recent report from the Russell Group: Realising Our Potential Backing Talent and Strengthening UK Research Culture and Environment.[3]
4.2.2. Recent initiatives from funders such as an increase in open access publication and data sharing policies, greater requirements for information on intended analysis and preregistration of protocols are welcomed and these should be harmonised across funders as far as possible to reduce the administrative burden on researchers and research support staff.
4.2.3. Funders should broaden out the desirable criteria for researcher evaluation to include commitments to good research practices such as open research initiatives, reproducibility, and contributions to the wider research community.
4.3. Role of research institutions and groups in addressing reproducibility “crisis”
4.3.1. Research institutions should ensure the provision of training and support for researchers. This should include discipline-specific material such as experimental design and statistical analysis, and broader topics including transparent reporting, open research, and authorship.
4.3.2. Research institutions should seek to champion and promote good research practices and professional responsibilities, as well as support researchers to self-correct where necessary.
4.3.3. In line with the role of funders, research institutions should broaden out their desirable criteria for researcher evaluation to include commitments to good research practices such as open science initiatives and contributions to the wider research community. These should be integrated into recruitment and promotion criteria and reflected in internal awards and funding schemes.
4.3.4. Research institutions should promote a culture of accountability with clear, transparent, and easily accessible processes for reporting and investigating cases of poor practice in research and in other cases of poor professional conduct such as bullying and harassment.
4.4. Role of individual researchers in addressing reproducibility “crisis”
4.4.1. Individual researchers play a key role in upholding good research practices, leading by example and promoting good research culture among colleagues. This may involve engaging in open research initiatives, undertaking, and providing training as well as peer support of colleagues whether mentoring, line management or more informal processes of support. However, there must also be a recognition of the limitations of the role of individual researchers given the time and resources available to them.
4.4.2. Researcher-led initiatives at King’s College London have included, among other contributions: the establishment of an Open Research Award scheme within the Institute of Psychiatry, Psychology & Neuroscience; the RIOT Science Club conference ‘Open Research: a vision for the future’ which was attended by over 300 participants; and the creation of an Open Research Calendar[4] in collaboration with external colleagues. These have clearly illustrated the power and effectiveness of peer-led initiatives and support.
4.5. Role of publishers in addressing reproducibility “crisis”
4.5.1. The recent increase in quality control checks on experimental design, methodology and analysis is welcomed. This can be in the form of self-reporting checklists or specialist editors to assess robustness.
4.5.2. Publishers should mandate open data and/or open materials or code where appropriate and encourage preregistration or explore alternative formats such as Registered Reports.
4.5.3. In line with the cultural shift required for valuing null or negative results, publishers should assess potential biases in their processes and ensure adequate opportunities for the dissemination of null results.
4.5.4. To align with the culture of accountability that should be promoted within research institutions, publishers should also ensure transparent and easily accessible routes for reporting and investigating research integrity issues. This should include processes and support for researchers who need to self-correct.
4.6. Role of governments and the need for a unilateral response /action
4.6.1. The UK Government should continue to set a high standard for engagement in this area, but a robust strategy will require integration with other funders nationally, and other legislatures internationally. cOAlition-S has demonstrated that trans-national agreements can bring about change in this sector.
4.6.2. Governments should examine how funding structures for research-intensive institutions such as Quality-Related (QR) research funding and other funding streams impact the ability of research institutions to provide career stability for researchers. Where new policies may lead to changes in funding streams, consideration should be paid to the impact on the research integrity landscape. For example, oversight should be maintained of the impact of increasing incentives for industrial or commercial collaborations on transparent reporting.
4.6.3. Policies designed to have a positive impact on public confidence in science should adopt an integrated approach to research funding. Beyond funding and resources for STEM disciplines, issues of reproducibility will need to be conceptualised, analysed, and addressed from a wide range of viewpoints and as such will require investment and engagement with researchers in all disciplines, including in the Arts & Humanities.
5.1. In addition to the schemes or initiatives already mentioned in this submission, we would welcome the provision of guidelines for researchers on conceptualisations of reproducibility and supporting this quality in their work.
6.1. We welcome the potential for the UK Committee on Research Integrity (UK CORI) to raise the profile of research integrity, good research practices, and initiatives and to help promote public understanding of these issues.
6.2. UK CORI can play a key role in the co-ordination of activity and strategy between stakeholders across the research landscape.
(October 2021)
[1] https://www.kcl.ac.uk/research/support/rgei/research-integrity/research-integrity-champions-and-advisors
[2] https://www.ukrn.org/local-network-leads/