Written Evidence Submitted by the National Centre for the Replacement, Refinement and Reduction of Animals in Research (NC3Rs)

(RRE0033)

The National Centre for the Replacement, Refinement and Reduction of Animals in Research (NC3Rs) is a UK science-based organisation primarily funded by the MRC and BBSRC to support the development and use of new approaches which replace, reduce and refine the use of animals in research and testing (commonly termed the 3Rs). Our activities are divided between our role as a funder of 3Rs research and innovation and programmes led by NC3Rs staff. Our work on experimental design and reporting is one of our longest running programmes and our main activity supporting the reduction in animal use. Our aim is to ensure that all primary research that uses animals is rigorously designed and transparently reported, increasing its reproducibility. This is important to the NC3Rs mission as it ensures that studies use the minimum number of animals consistent with the scientific aims, and that findings are robust and truly add to the knowledge base thus avoiding animals being used unnecessarily, including in subsequent experiments that build on, or aim to reproduce, the original findings.

 

  1. We are pleased to have the opportunity to respond to the Committee’s consultation. We have focused our reply on factors and actions that are most relevant to the use of animals and therefore the NC3Rs mission. Our response is not intended to imply that animal research suffers from reproducibility problems to a greater extent than other areas of research. The lessons we have learned and the principles and processes we have established could be readily adapted to aid the problem in other fields of research. Indeed, we have recently started a new initiative to develop reporting principles for in vitro research (working title RIVER – Reporting In Vitro Experiments Responsibly).

The breadth of the reproducibility crisis and what research areas it is most prevalent in

  1. Over the past 15 years, we have focused on improving the quality of animal research. Evidence suggests that the majority of publications reporting animal research cannot be reproduced. This hampers the translation of preclinical research findings to humans and wastes animals. We first raised awareness of issues with the quality of animal research in 2009 – before the term ‘reproducibility crisis’ was coined – and provided evidence of systematic flaws in the way animal experiments are designed and reported in the scientific literature. It is now widely recognised that issues around the rigour and transparency of research are major contributing factors preventing the reproducibility of animal and preclinical research.

The issues in academia that have led to the reproducibility crisis

  1. Various issues contribute to the lack of reproducibility of animal research. There is limited statistical support available for researchers and many universities do not have statisticians to help researchers design and analyse their experiments. Where training on experimental design and statistical analysis is available to researchers, the training is often not tailored to tackle issues that are specific to animal experiments. As a result, researchers often have a poor understanding of the importance of rigorous experimental design and the implications of the statistical tests used to analyse the results. Since the scientific cycle is primarily driven by review by scientific peers, these issues are not addressed, rather they are perpetuated as those assessing funding applications, conducting ethical review of the protocols, and reviewing and editing the research articles before publication have themselves not received appropriate training in experimental design and statistics.

 

  1. A “publish or perish culture” exists in academia. Researchers are generally recognised and rewarded (e.g. in recruitment and promotion) for publishing articles in journals with high impact factors. Typically, these journals prioritise novel, unusual and exciting findings over rigorous methodologies. Despite many funders providing guidance (e.g. as signatories of DORA – the San Francisco Declaration on Research Assessment) and the Research Excellence Framework making it clear that impact factors and journal-based metrics should not be used as a surrogate measure of the quality of the research, most researchers report experiencing pressure to publish in high impact journals. This incentivises a range of questionable practices that accelerate the research and increase its publication potential but reduce the likelihood that it will be reproducible. Such practices include the use of small sample sizes in underpowered experiments, failure to use measures that reduce the risk of subjective bias, excluding outliers without predefining exclusion criteria, pseudoreplication, selective reporting of ‘interesting’ results, HARKING (that is hypothesising after the results are known), p-hacking (that is running multiple analyses on the dataset and artificially increasing the likelihood of finding a statistically significant result purely by chance), and incomplete reporting in the published article that prevents a full assessment of the reliability of the findings (including hindering independent replication by other researchers).

 

The role of the following in addressing the reproducibility crisis

Research funders, including public funding bodies
 

  1. Funders have a major role to play in improving the quality of animal research, including its design, conduct, analysis and reporting. In recent years, based on our work funders such as the MRC have increased their expectations on the level of detail on experimental design required in grant applications in order to help address concerns about rigour and reproducibility. Grant applications involving the use of animals are now required to include a description of the number of experimental and control groups, the sample size along with justification for the number of animals to be used, and measures to minimise subjective bias such as randomisation and blinding. Aligned with this, we have run training workshops for panel members from a range of funding bodies to help members recognise the hallmarks of rigorous research. Most of our own funding panels also now include statisticians, specifically tasked with reviewing the experimental design and analysis in the grant applications we receive to ensure the quality of the 3Rs research we fund.

 

  1. Many funders nationally and internationally encourage the use of resources we have developed to promote robust and reproducible animal research. This includes the ARRIVE guidelines and the Experimental Design Assistant (EDA). The ARRIVE guidelines set out the minimum information that should be included in any publication reporting the use of animals. They are accompanied by a document which provides further information about each of the 21 items in the guidelines, including the rationale and supporting evidence for their inclusion in the guidelines and examples of good reporting from the published literature. 

 

  1. The ARRIVE guidelines are endorsed by over a thousand journals and have been highly cited (8,000 citations to date). Organisations such as the Academy of Medical Sciences in the UK and the National Academies of Sciences in the USA have highlighted the importance of the ARRIVE guidelines for improving the reproducibility of animal experiments. More than 30 funders, including Cancer Research UK and the Australian National Health and Medical Research Council, have adopted the guidelines and in some cases embedded their use in grant terms and conditions. In the USA, the ARRIVE guidelines have been included in the Principles and Guidelines for Reporting Preclinical Research published by the National Institutes of Health (NIH) in 2014. More recently, in June 2021 a report on enhancing rigor, transparency and translatability in animal research produced by the NIH’s Advisory Committee to the Director recommended that the NIH should expect all vertebrate and cephalopod research to include the ARRIVE 2.0 Essential 10 at the publication stage. Further information on the ARRIVE guidelines, including resources to support their use (e.g. translations in nine languages) can be found on the ARRIVE website.

 

  1. The EDA is a unique system that uses computer-based logical reasoning to help researchers design robust animal experiments and address common pitfalls particularly relating to the internal validity of the study. The EDA includes an app which provides researchers with bespoke feedback and advice on their experimental and analysis plans, and a website with detailed information on experimental design. It also includes functionality for minimising bias through randomisation and blinding, for sample size calculations, and for choosing the appropriate statistical analysis methods. There are over 11,000 registered users of the app worldwide, and currently 24 funding bodies recommend use of the EDA in the preparation of grant applications – these include Wellcome, the NIH, the National Science Foundation and the German Federal Ministry for Education and Research.

 

  1. Funders also have a role in encouraging the publication of all research outputs, including negative results and methodologies. To support this, we and funders such as Wellcome have set up dedicated publication platforms (the NC3Rs gateway and Wellcome Open Research respectively). These platforms promote the rapid dissemination of research as the articles are published before peer review and assessed based on rigorous methodology rather than the direction or novelty of the findings.

Research institutions and groups
 

  1. There is more that research institutions could do to ensure animal studies lead to reproducible and reliable findings. This includes providing researchers with access to support from statisticians specialising in animal experiments, from the early stages of the experiment, so that studies are designed, conducted, analysed and reported in a rigorous manner. This represents a significant financial investment and is a long-term goal as there are relatively few individuals with such expertise. It is important that nationally this skills gap is addressed.

 

  1. There should be an institutional expectation that researchers will use the resources that are freely available, including those that we provide such as the EDA and the ARRIVE guidelines. We have recently published a dedicated Action Plan for institutions to help support the use of the ARRIVE guidelines. The Plan sets out a series of activities – divided into three tiers by ease of implementation and degree of impact – that institutions can undertake to integrate ARRIVE into their policies and practices.

 

  1. Institutions should also make better use of their animal ethics committee (in the UK the Animal Welfare and Ethical Review Body – AWERB) to implement policies promoting reproducibility. Although AWERB members themselves typically have limited expertise in experimental design, they should encourage and expect researchers to use the available resources designed to improve experimental design, such as the EDA and the ARRIVE guidelines. Additionally, although the ARRIVE guidelines focus on reporting, the items that make up the guidelines also represent a useful checklist of information that should be considered during the ethical review of research using animals. AWERBs can make use of ARRIVE by requesting researchers provide information in line with the specific items of the guidelines during ethical review of their project licence applications. This is a practical way to ensure that experimental design concepts have been fully considered during study planning, and that all the information required to report a study well is recorded during its conduct.

Individual researchers
 

  1. Researchers often have limited access to statistical support for animal studies and tend to rely on historical precedent rather than best practice to design, conduct and report their experiments. They also operate within a competitive system with high workloads, unsecure employment, and pressure to produce results and publish them quickly. These factors have an impact on how animals are used and the quality of the science generated from them (e.g. studies using too few animals to be reliable because numbers were based on historical precedent rather than a sample size calculation). The system needs to allow researchers time to attend training in experimental design, analysis and reporting that will make their experiments more reproducible, and more time needs to be allocated to implement the training in practice as conducting more rigorous science can take longer, putting researchers at a disadvantage with current career incentives.

 

  1. The scientific literature is a vital source of information but is subject to significant shortcomings as highlighted above. The most effective and impartial way to examine the quality of the literature is through systematic review and meta-analysis. For example, systematic reviews can highlight areas of weakness in experimental design and reporting driving improvements in future studies. Systematic reviews can also provide the most unbiased way of choosing an animal model and making informed decisions about sample sizes as well as investigating sources of heterogeneity which are not evident in individual studies, but become obvious in a systematic review (e.g. different treatment effects in male and female animals). To assist researchers to carry out systematic reviews we have funded SyRF, a free-to-use online systematic review platform developed by the Edinburgh-based CAMRADES group and we continue to support its development and maintenance on an ongoing basis.

Publishers

 

  1. Publishers and journals have a role in ensuring that the publication record is transparent and accurate. Most journals endorse reporting guidelines, such as ARRIVE, but few use them in practice or actively enforce reporting standards. To improve the reporting of animal research, we have developed a range of resources associated with the ARRIVE guidelines that journals can use to embed the guidelines into their editorial processes and ensure that the experiments in research articles are described comprehensively and transparently. This transparency will enable readers and reviewers to scrutinise the research adequately, evaluate its methodological rigour, and reproduce the methods or findings. We have described strategies that journals can use to improve the reporting of animal research in a dedicated Action Plan which sets out a tiered list of activities such as promoting examples of good practice to authors and reviewers; requiring an author checklist indicating where in the manuscript relevant information on experimental design can be found; providing training for journal staff highlighting the importance and benefits of transparent and complete reporting; and checking adherence with reporting guidelines and requesting any missing information before the manuscript is published (e.g. by using the ARRIVE compliance questionnaire that allows for a streamlined evaluation).

 

  1. A further role for journals is to ensure that the manuscripts they publish are methodologically sound and that the findings are reliable. This can be achieved by systematically involving statisticians in the peer review process to check that the design and analysis of the experiments described in a manuscript are adequate to support the conclusion and inferences of the study. Journals should refrain from setting expectations (implicitly or explicitly) that findings from in vitro experiments should routinely be supplemented or confirmed by animal studies in order for a manuscript to be published (e.g. it is common for reviewers to ask for animal studies to be conducted). Reproducibility relies on carefully planned and well-reported research – requiring animal studies in a short time period to support publication risks decreasing rigour when planning or executing the study.

Governments and the need for a unilateral response / action

 

  1. All Government bodies engaged in animal research should require the use of the EDA and ARRIVE guidelines.

 

What policies or schemes could have a positive impact on academia’s approach to reproducible research?

  1. Academic researchers would benefit from training schemes on experimental design, statistical analysis, transparent reporting and reporting guidelines, and systematic reviews of animal studies. To support this, we are developing e-learning resources to enable self-directed learning on key topics in experimental design, such as blinding, randomisation and sample size calculation. As part of the groundwork for the e-learning resource on blinding we are collaborating with AstraZeneca on a project identifying barriers to implementing blinding and effective strategies to overcome these.

 

  1. Statistical support dedicated to animal research should be available to all academic researchers. Such support is often available to researchers working in the pharmaceutical industry but many academic institutions in the UK do not offer help, or the support is limited. For example, support is provided by a statistician not specialising in animal research and not familiar with common issues that need to be addressed in this type of experiment, or support is only available to senior researchers at specific time points (e.g. as part of the application process for a five-year project licence application to use animals) whereas most experiments are designed and conducted by early career researchers, who need an in-depth review of the experimental and statistical analysis plans before the start of each experiment. Use of the EDA and ARRIVE guidelines is still not universally adopted by academic researchers and many funders and publishers use terms such as “endorse” “recommend” or “encourage” when referring to them. A more active approach needs to be taken across the sector to enforce policies and practices and generate a culture in which the choice of conducting high quality well designed and reported animal experiments is not optional.

 

  1. Most research using animals will be reviewed on three occasions – by the grants panel, the AWERB, and the regulator for scientific procedures on animals. This scrutiny is failing if animal research is poorly designed, conducted, analysed or reported, raising serious concerns about public assurances on this sensitive and often controversial area of science. AWERBs and regulators have a key role in ensuring and enforcing local standards. This is as much about ethical considerations as it is scientific quality. Poor science wastes animal lives – bodies with responsibility for ethical review such as AWERBs and regulators (the Animals in Science Regulation Unit in the UK) should take into consideration factors relating to reliability, rigour and reproducibility in order to make a proper harm-benefit analysis on whether the research and associated animal use and suffering are justifiable. The Animals in Science Committee has similarly highlighted this point and it is important that it is reflected in policy.

 

How establishing a national committee on research integrity under UKRI could impact the reproducibility crisis.

  1. We do not consider that the national committee would have a direct impact on the reproducibility of animal research per se. However, its wider impacts on research culture could ultimately lead to changes that facilitate improvements across all areas of science and a more receptive response to efforts to improve research quality.

 

 

(September 2021)