Written Evidence Submitted by the Association of Medical Research Charities

(RRE0074)

Introduction

The Association of Medical Research Charities (AMRC) is a membership organisation of the leading medical and health charities funding research in the UK. We represent around 150 medical research charities including the Wellcome Trust, Cancer Research UK and the British Heart Foundation as well as smaller charities who invest in rare diseases and specific areas of unmet need. In 2020, out of a total charitable investment of £2.9 billion by AMRC members, £1.7 billion went towards research in the UK.[1] We are pleased to be able to contribute to the Committee’s deliberations on this topic. 

The importance of research integrity

Research integrity is important to ensure confidence and trust in the methods and the findings of the research. It relates both to the scientific integrity of research undertaken and to the professional integrity of researchers.  Where integrity of either one is questioned, this can cause the research outcomes to be questioned and the value of scientific expertise to be undermined. Poor reproducibility threatens the reputation of biomedical science and the public’s trust in its findings, and particular in medicine, confidence in the use of treatments.

A few incidents which gain significant publicity can destroy trust and trust is very difficult to rebuild.  For example, the fraudulent claim that the MMR (measles, mumps and rubella) vaccine causes autism received a lot of publicity through mainstream media and gained traction across the Internet and has had a long-term impact on vaccination rates[2].  When discussing irreproducibility in research, we should distinguish between deliberate fabrication or falsification of data which is rare[3], and unsatisfactory research practices that can compromise research integrity.

However, the COVID pandemic has increased the visibility of science and it is important that this is correctly harnessed by the life sciences sector and the media to ensure greater involvement, engagement and understanding of the public in science. For charity funders driven by public donations, upholding research integrity is important to ensure that the public and patients can have confidence in research so they can benefit from it.

The breadth of the reproducibility crisis

Despite the ambition for high research integrity, over recent years there has been a sustained concern within academia over reproducibility and replication, which some argue bring integrity into question[4],[5]. For example, in 2016, when surveyed two-thirds of a sample of researchers from around the word (n=1,576) said that current levels of reproducibility are a major problem[6]. On the other hand some question the narrative of a ‘reproducibility crisis’, with suggestions that it can be counterproductive by fostering ‘cynicism and indifference’ in younger researchers, and ‘risks discrediting the value of evidence and feeding antiscientific agendas’. [7] There is a need to clarify the distinction between reproducibility, replicability and repeatability to ensure the sector is clear what it is aiming for.

There are many underlying causes that lead to irreproducibility in research.  These include poor experimental design, poor statistical practices, inadequate reporting of methods, and problems with quality control[8] and deeper rooted cultural issues as outlined below. Reproducibility is a challenge not just for biomedical research, and different fields may be able to learn lessons from one another[9].

It is difficult to quantify the exact level of irreproducibility in the published literature. A 2015 meta-analysis of past studies regarding the cost of non-reproducible research estimated that $28 billion per year is spent on preclinical research that is not reproducible[10]. The current level of waste in health and medical research is contentious, but one estimate puts it at over 85% of the nearly $200 billion annual global spend.[11] It has been shown that waste occurs at four successive stages during the research process: the choice of research questions; the quality of research design and methods; the adequacy of publication practices; and the quality of reports of research[12] (Figure 1).

Figure 1: Stages of research waste

Source: The New Power University: The social purpose of higher education in the 21st century, Jonathan Grant (2021)

These ‘bottom-up’ problems are worsened by the current research culture and ‘top-down’ influences including the time and financial pressures of science, poor training, and reward and recognition structures focusing on results, novelty and publication.[13],[14]  Therefore there is a need for mechanisms and initiatives to drive a culture that incentivises integrity. Greater understanding is also needed around the process of research, and the evolution of knowledge to appreciate the role and value of evidence. It is also important to ensure that the community responds proportionately to allegations of irreproducibility. The research community must foster a ‘no-blame’ culture that encourages researchers to identify factors and practices that led to the discrepancy and make corrections. Self-regulation by the research community must be encouraged.

Initiatives and activities to improve reproducibility

Funders, publishers, institutions, researchers and others all have a role to play in reducing waste and improving reliability of research[15]. For example, funders can support their researchers and influence behaviour through the provision of infrastructure and support, by ensuring research assessment is undertaken in a meaningful way using a broader range of assessment criteria, or by making funding contingent on researchers' following certain good practices.

Many charities exist because of donations from the public and therefore they have a duty to fund research that will be valuable to patients, their families and carers. As such, the responsibility for charities to support the best quality research that is reliable and repeatable is a driving force behind how they operate. Throughout the process of awarding funds, charities go to great lengths to ensure they prioritise funding towards the most pressing questions, have robust mechanisms of awarding funding to only the highest quality research proposals and investigate new ways to measure impacts that really matter to patients and the wider public. Some charities also take research reproducibility into account in their strategic priorities and impact plans (see case study: Autistica).

In recent years, funders have made substantial efforts toward implementing changes in policy and practice aimed at improving reproducibility. Below are a few examples of how charity funders are contributing to efforts to improve reproducibility and reduce research waste through: asking the right research questions, encouraging robust methodology and research design, ensuring the research results are reported and accessible, and funding proof-of-concept and reproducibility studies.

Asking the right research questions

All AMRC charities have a published research strategy that clearly outlines the areas of research where they intend to focus their resources. Charity-funded research is driven by patient priorities. 83% of AMRC charities use patient voice in their research strategy or influencing work, ensuring that funding is directed where it will make the most difference to patients and leading to more efficient development of treatments. For example, in August 2021 Epilepsy Research UK launched the SHAPE Epilepsy Research Network to build the biggest ever community of people affected by epilepsy to influence and shape future research. This will make studies more effective, relevant and cost-effective.[16]  The patient voice is important for increasing value and reducing waste. To date, more than 40 AMRC members have been involved in Priority Setting Partnerships (PSPs) over the years which set patient-led research priorities that will bring tangible benefits to people in need (see case study: Marie Curie).

Encouraging robust methodology and research design

Improving research design and methodology is vital to enable reproducibility of studies. AMRC members are strongly encouraged to promote existing guidelines in their grant terms and conditions, such as the Animal Research: Reporting of In Vivo Experiments (ARRIVE) guidelines[17] from the National Centre for the Replacement, Refinement, and Reduction of Animals in Research (NC3Rs)[18]. Continuing education, training and mechanisms for recognition may be one way to address specific issues related to irreproducibility. Some funders strongly encourage host organisations to provide PhD students and postdoctoral researchers with training in statistics, experimental design and good research practice[19].

Ensuring the research results are reported and accessible

Over the past 30 years there has been a push towards open science, making results and publications publicly available – particularly those funded by public money.  Benefits include accelerating discovery and innovation, increasing reuse and preventing duplication. It also allows findings to be reproduced and challenged. The majority of AMRC member charities have clauses in their terms and conditions of grant that expect publication of research findings. AMRC member charities also support initiatives that allow unrestricted access to published research. Many are members of Europe PMC, and some have their own open access policy and guidance to researchers on what open access costs they cover[20][21].

In recent years, some charity funders have developed and launched open research platforms (see case study: AMRC Open Research).

Funders use their policies and terms and conditions to ensure trial results are reported and the research findings are freely available, including null and negative results[22]. The Health Research Authority’s (HRA) new monitoring system is being seen as a huge step towards clinical trial transparency as it aims to monitor whether clinical trials in the UK make their results public as part of a national strategy for ensuring that all clinical trials are registered and report their results[23]. The monitoring system was developed as part of the wider #MakeItPublic strategy, which AMRC fed into member views, calling for a greater focus on measures to encourage registration, reporting of results, and feedback to participants.

Funding proof of concept and reproducibility studies

Quality of research is the most important aspect regardless of whether the outcome is positive or negative. Many charities fund studies to generate pilot or proof-of-concept data to help determine whether an idea warrants subsequent major funding[24]. This mechanism of supporting research allows validation of feasibility, and brings benefit to the charity and wider life science ecosystem by taking on high risk areas that the charity or others can subsequently fund once initial research is conducted. Some charities also offer support for replication efforts (see case study: Parkinson’s UK). The key to replicability is transparency and complete reporting.

There is a need to work across the system

AMRC welcomes the creation of the UK Committee on Research Integrity (CORI), in response to the 2017 Research Integrity inquiry from the Select Committee[25]. We are pleased that UK CORI will champion research integrity in the UK, working closely with the sector, to develop, identify and share good practice[26]. We understand that UK CORI plans to take a broad view of the meaning of the term ‘integrity’ to include robust and reliable research, good governance, good training, the research environment as well as appropriate handling of cases of misconduct. This is welcome and we hope that UK CORI will involve itself on wider issues of irreproducibility that do not constitute misconduct.

We look forward to working with CORI as the Committee gets going with its activities related to research integrity and reproducibility. For example, developing and setting the standards and guidance that research institutions and funders can be expected to be held to when they are investigating cases of misconduct. Care should be taken to ensure any new measures to improve integrity do not unnecessarily increase the bureacractic burden on the sector or stifle creativity. Clarifying the key responsibilities of different groups may be helpful to provide tangible objectives and to prevent each group being overwhelmed with the number of changes.

Due to the range of influences in the system, there is a need to work across the system to deliver change.  A few years ago, AMRC became a supporter of the Concordat to support research integrity[27]and our members, Wellcome and Cancer Research UK (CRUK) are signatories.[28] CRUK and Wellcome are also members of the UK Reproducibility Network (UKRN) External Stakeholder Group[29]. UKRN is a national peer-led consortium that aims to ensure the UK retains its place as a centre for world-leading research. UKRN does this by investigating the factors that contribute to robust research, promoting training activities, and disseminating best practice. CRUK has funded UKRN training courses such as Data Management Skills for Open Science.[30] CRUK also have research integrity advisers in two of their institutes (see case study).

Overall, whilst work is underway there is still more to be done to improve transparency and reproduciblity which will reduce waste and increase public confidence.  This is an issue that must be approached with multi-stakeholder involvement and drawing on best practice internationally. Given the global status of research, it must be acknowledged that changes made in the UK which shift the reward and incentive structures may need to be also adopted internationally to avoid tension for researchers with global connections. 

(September 2021)