Written Evidence Submitted by the British Pharmacological Society

(RRE0054)

About us

The British Pharmacological Society (‘the Society’) is the primary UK learned society concerned with research into drugs and the way they work. The Society has around 4,000 members working in academia, industry, regulatory agencies and the health services, and many are medically qualified. The Society covers the whole spectrum of pharmacology, including laboratory, clinical, and toxicological aspects. Pharmacology is a key knowledge and skills base for developments in the pharmaceutical and biotech industries, and is therefore fundamental to a thriving UK industry and R&D. These skills allow members of the Society to identify therapeutic areas of clinical need, develop novel treatments that target these areas and ensure these new treatments are incorporated into healthcare practice bringing benefit to patients. The Society publishes three scientific journals: the British Journal of Pharmacology, the British Journal of Clinical Pharmacology, and Pharmacology Research and Perspectives[1].

Executive summary

The Society supports the House of Commons Science and Technology Committee’s aim to further investigate research integrity through an examination of research reproducibility. We welcomed the results of the 2017 enquiry which resulted in the government asking UKRI to form a national research integrity committee. In July 2021, UKRI reaffirmed its commitment to host this new committee[2]. We agree that there is more to be done in the reproducibility sphere to ensure that research is robust and transparent and can reliably inform future discoveries.

Specifically, we recommend:

• That the Committee advises all publishers develop guidance on reproducibility and transparency, similar to that of our Society journals, the British Journal of Pharmacology (BJP)[3], British Journal of Clinical Pharmacology (BJCP)[4] and Pharmacology Research & Perspectives (PR&P)[5].
• Learned societies have important, direct relationships with researchers and educators, and are therefore well-placed to drive and support ambitions. UKRI should explore opportunities to collaborate with learned societies to leverage such impact. For example, the Society has worked with The Physiological Society to establish the Research Animal Sciences Education Scheme (RASES)[6], which aims to improve consistency and standards in the use of research animals – reproducible data from well-designed experiments is critical for translation of findings to humans. Focusing resource on educational interventions can help improve research reproducibility for the long-term.
• The Society is concerned about the changes to Home Office inspections and believe that limiting interactions between researchers and the inspectorate may have a detrimental effect on the quality of science. We therefore urge the government to mitigate unintended consequences by continuing to facilitate information sharing and interactions as part of this new way of working.
• Further focus and investigation into this area, by the government and relevant bodies must be a priority, as lack of research reproducibility continues to be damaging for science[7]. We highlighted some of the issues in our submission to the committee’s previous enquiry on research integrity[8] e.g., poor research culture and a ‘pressure to publish’ are key drivers when it comes to the main issue of ‘inadvertent errors or questionable research’. To solve the issues, we must address culture, incentives, and education[9],[10]. However, any guidelines and requirements that are put in place must not increase bureaucracy for the individual researcher.

Our Submission

The Society’s response focuses on what needs to be done to improve research reproducibility and what we are doing to mitigate the issues.

We will specifically highlight:

-         The background and some of the key issues for the Society and its members.

-         The work we as journal publishers are doing to address the issues.

-         The emphasis we put on reproducibility when it comes to education and training.

1. Background and key factors

1.1             The research reproducibility “crisis” has been described as a significant methodological issue caused by an inability to replicate or reproduce some scientific studies leading to distortion of the scientific literature, thus rendering much of it inaccurate. In a 2016 survey[11], coordinated by Nature, more than 60% of respondents said that pressure to publish and selective reporting always or often contributed to problems in reproducibility. Other issues highlighted include poor oversight, low statistical power and variability in reagents or the use of specialised techniques that are hard to duplicate. Furthermore, about 90% of respondents felt that there are negative impacts related to reproducibility. In that survey, the majority of respondents agree there is an issue. Conversely, Fanelli argues[12] that the “crisis” narrative is somewhat unhelpful and to an extent, incorrect e.g., the reproducibility predicament is not so bad that much of the scientific literature is inaccurate. He goes on to suggest that the way in which reproducibility concerns can be combatted are through empowerment of early-career scientists. He argues that those beginning their careers will be disheartened if researchers continually hear the narrative that “science is in crisis”. A recent commentary in Nature[13], by the coordinators of the Brazilian Reproducibility Initiative, suggests that alongside making the average paper more rigorous, it might be more feasible to improve systematic confirmation of preliminary findings. Furthermore, one way to tackle the reproducibility issue is by placing the burden of replicability on the community, not on individual laboratories.

1.2             In our previous response to the Committee’s research integrity inquiry, we highlighted that research culture and a ‘pressure to publish’ are key drivers when it comes to the main issue of ‘inadvertent errors or questionable research’. With the creation of initiatives and groups like the UK Reproducibility Network (UKRN)[14], a national peer-led group association that aims to ensure the UK retains its place as a centre for world- leading research, it is clear the sector is working together to tackle the current issues. Alongside this, there are new online technologies paving the way to tackle problems of protocol[15]. However, there is still a problem as highlighted by the need to call this inquiry and the continuing evidence of questionable studies showcased by databases such as Retraction Watch[16].

1.3             While we are choosing to focus our response on the role of publishing and the role of education and training, it is important to note that this is a broad issue across the scientific and academic spectrum that must be addressed. It is also crucial to highlight that there are others in the sector who have responsibility, including funders, the government and research institutions. Funders of research must have processes in place to ensure that work is carried out to the highest levels and published to open access standards. The introduction of initiatives such as Researchfish[17] are an attempt to do this. However, whilst the collection of outcomes data is important the funders have placed the burden of reporting again at the feet of the academic. The experience of researchers that we have heard from would suggest that such initiatives are cumbersome, and we would suggest that greater thought on behalf of the funders is needed to address the issues in a shared manner that does not lay the entire burden with the academic. The Society view is that reporting into programmes like Researchfish should be centralised: most, if not all, researchers have an ORCID ID[18], therefore, data should be able to be transferred from that source so there is not an unnecessary focus on administration and bureaucracy, at the expense of research.

1.4             The Society is concerned about the broad of issue of inconsistency in reagents and other materials and biologics used in research. The problem with antibodies is well-recognised, but it also ranges from components included in culture media to cell lines and mouse strains. These are sold to research institutes, universities, and other places where research is undertaken. Some distributors of research tools, particularly with antibodies and reagents, do not always carry out the appropriate evaluations and validations. This means that these vital tools used in research may not always be of the appropriate quality or have been properly validated, such that reproducing a published study using some reagents becomes problematic. In some instances, the companies supplying reagents buy in their reagents (such as antibodies) selling them on to researchers without checking the quality of the product. This leaves the research community to carry out the validation themselves, wasting valuable time and effort. We believe there needs to be further work to ensure accurate validation and quality control, particularly of reagents such as antibodies, before they can be made available to the research community. Some journals e.g., the BJP, request that authors provide a Research Resource Identifiers (RRID) designation[19], which goes some way to mitigating this issue and other publications should be encouraged to follow this lead. Improvements in openness, by all involved, would also help.

1.5             From cells to populations, good experimental design and sufficient powering of experiments are other factors that need to be improved across the research landscape. For example, when a researcher is doing histology, it is easy to focus only on a very small aspect of the tissue which could lead to bias. It was also highlighted to us that there is concern that there is often inadequate blinding undertaken in non-clinical work, to the level usually associated with many clinical trials of medicines which may lead to poor translation of preclinical work into the clinical setting. The ARRIVE 2.0 guidelines[20] are important for providing guidance on this issue – the BJP published an editorial about the refreshed guidance[21]. Although these guidelines are predominantly focused on the use of research animals many of the principles within them are applicable to human cells and tissues e.g., a researcher experimenting on stem-cell derived human organoids will face the same issues as a researcher experimenting on mice. Further information about resources the Society has created in relation to experimental blinding can be found in sections 3.3 and 3.4.

1.6             The Home Office has recently changed the way in which it will be inspecting and providing project licences for animal research[22]. Previously, the Home Office would do routine visits to institutions, but they have now stated this is not how they will be determining license criteria – there may be unintended consequences of this change. One of the things the inspectorate does is come to a researcher’s institution and discuss the project license with them including problems they may have encountered. If the inspector has a spectrum of knowledge about research going on at other institutions nationally – and many of them do – they often suggest to the researcher that they collaborate or contact those doing research elsewhere so their techniques or suchlike could be improved to come in line with what the inspectorate considers best practice e.g., post-surgical pain relief or best practice in anaesthesia. While this may be beyond the scope of this specific consultation, there is potential in 3-4 years’ time that this will impact reproducibility because the same knowledge sharing is not occurring. The Society is concerned about the changes to Home Office inspections and believe that limiting interactions between researchers and the inspectorate may have a detrimental effect on the quality of science. We therefore urge the government to mitigate unintended consequences by continuing to facilitate information sharing and interactions as part of this new way of working.

2. The role of journals

2.1             Publishers play a key role in ensuring those submitting to their journals and publications aim to produce reproducible research by encouraging transparency and honesty. In 2019, Hrynaszkiewicz[23] highlighted the practical approaches undertaken by many publishers in an effort to ensure high-quality reproducible research appears in their publications. The expectation for researchers to publish in peer-reviewed journals, thereby validating their work, means journal policies are a powerful tool to influence research practice.

2.2             In our 2017 submission to this committee, we argued that research funders and publishers have an important role to play in ensuring their standards of rigour and assessment support such practices. This remains to be the case and we reiterate this statement.

2.3             The Society and the American Society for Pharmacology and Experimental Therapeutics (ASPET) work in close collaboration, both have been actively involved in increasing the reliability of biomedical research generally and ensuring transparency in publications specifically. A review from Jarvis and Williams[24], highlights the perceptions, uncertainties, and knowledge gaps regarding the reliability of biomedical research outcomes in preclinical research. Whilst the review highlights initiatives such as remediation activities improve transparency and consistency, it noted there are still issues when it comes to improving the historical process of independent replication in validating research findings.

2.4             When publishing papers, all the materials and methods must be given in sufficient detail to allow another researcher to reproduce the work. There will, however, occasionally be some variables which are difficult to describe or control and for which not much can be done e.g., animals kept in different locations, even if genetic background is maintained there may be a change of gut microbiota or of diet which can lead to different phenotypes. Many studies published in high impact journals can turn out not to be reproducible. In fact, a recent project to replicate 50 high-profile cancer papers had to be reduced to 18 because of the uncertainty about various aspects of the studies protocols[25]. This could be because the analysis was underpowered or the number of repetitions within the experiment were too low. Chance events can lead to incorrect conclusions which is why it is necessary to have sufficient numbers and statistical power in the experiment. Further, papers in many journals have insufficient depth of recording protocols and (importantly) justifying choices of alternatives (e.g., reagents, instruments, plastics manufacturers, methods) to allow an appropriately trained follower to reproduce the experimental approach. Journals should attempt to find ways to ensure thorough methodology, recording protocols and choices of alternatives are included on paper submission.

2.5             There is vital role for peer review in adding value to publications, although it is not infallible. Assimilating critical analysis and the role of reviewers in improving the practice and presentation of science is something that may benefit from being embedded in research education & training through a   spiral learning process occurring at multiple levels of progression. However, such translatable skills are less easy to quantify than knowledge-based assessments, and so they often receive less emphasis. As we highlighted in our 2017 research integrity response to this committee: higher education institutions could look to instil better awareness of these issues and a ‘culture of integrity’ at the earliest stages of an individual’s research career.

2.6             In 2018-2020, the BJP launched guidelines and checklists[26] for authors submitting to BJP, to raise the reproducibility and transparency of published research. This is a crucial initiative because it encourages authors to develop accurate research. If they do not, they would be unable to publish in a highly acclaimed journal. The BJCP and PR&P also have similar guidance for ensuing openness of research and transparency. It will be important for journals across the board to develop similar policies so that it becomes harder to get dubious studies published. We recommend the Committee advise all publishers develop guidance for authors similar to that of the BJP, BJCP and PR&P.

2.7             In conjunction with journal-specific guidelines, other programmes which further engage early-career researchers with the journal and publishing process – and the issues of reproducibility within them - should be encouraged. For example, ReproducibiliTEA[27], an initiative that helps researchers create local Open Science journal club at universities to discuss ideas including how to improve science and reproducibility. The importance of education and training in this space will be further discussed in section 3.

3. The importance of education and training

3.1             Education and training are of the utmost importance in developing a scientist’s academic skills. One way to deal with concerns about reproducibility is to give researchers appropriate data management and analysis training as early as possible. If this is done, they have the ability to transfer skills when needed - most researchers will have to deal with a large dataset during the course of their career[28], thus the correct training is crucial. Further to section 1.5, it should be reiterated that researchers are not being trained in sufficient depth in appropriate experimental design and analysis. There could be several reasons for this including that universities are focusing more on high throughout testing [computer-based methods], particularly in the early years of science degrees. This is compounded by the external pressures to generate more enticing headlines and press-worthy research. This then leads to the temptation from some to cherry pick data and experimental outcomes, which may emphasise a less common/reproducible aspect of the work.

3.2             The Society supports the integrity and quality of the research undertaken by its members by hosting research meetings and workshops and providing educational material for the wider researcher, educator and student community. To maintain the integrity of research in all fields it is important that bioscience students are taught how to design, perform and report their own research studies as well as critically interpret and appraise published research. An example this is important for – but not restricted to – is research using live animals where reproducibility of research between labs or translatability of research to humans are critical to the development of safe and effective medicines. Unfortunately, as mentioned, there are issues related to the reproducibility of research and bench to the bedside translation of animal research, which in some areas, has been poor. Therefore, we have selected this as a case study where the Society has made efforts to mitigate some of the paucity in the education and training space.

3.3             The Society has been at the forefront of initiatives to ensure that the use of research animals in the UK is performed to the highest standard. We recognised that an important way to address issues around reproducibility and translatability was to support education, better equipping the next generation of the biomedical workforce with key knowledge that can in turn support future biomedical innovation. In the early 2000s, surveys by the Society and The Physiological Society[29] showed that the ability of UK universities to teach students and researchers the appropriate use of research animals was becoming severely limited by a lack of academic staff with the necessary experience and expertise, and a report[30] from ABPI and the Biosciences Federation highlighted sector skills gaps for in vivo sciences. To address this, the Integrative Pharmacology Fund (IPF) was established with finance provided by the pharmaceutical industry. It was intended to address the need for in vivo training and was supported by contributions totalling £4 million from AstraZeneca, GlaxoSmithKline (GSK) and Pfizer. A strategic decision was made by the steering group that the IPF would work with national funding bodies (the Biotechnology and Biological Sciences Research Council, BBSRC; the Medical Research Council, MRC; and the Higher Education Funding Council for England, HEFCE) to support in vivo education and training. This collaborative approach resulted in the £4 million contributed to the IPF leveraging total support of £22 million for in vivo research, education and training. An independent review and evaluation[31] of this funding initiative (published in 2016) demonstrated its successes but also highlighted the inconsistent level of teaching for this topic across the university sector as a whole. In response, the Society launched a ‘Research Animals Curriculum’[32] consisting of key learning outcome statements developed by experts from industry, academia, the NC3Rs and animal welfare organisations. Importantly, the curriculum is intended to support consistency in the education of all bioscience taught undergraduate and taught MSc degree programmes. Specifically where students are expected to analyse literature and/or data that have been generated from studies involving animals and extends beyond pharmacology. Whilst many universities already cover some of these learning outcomes within existing courses, there are only a few universities currently providing in depth education in this important area. Additionally, lecturers from some institutions may only have a limited personal experience of using research animals and may feel not feel confident in delivering the curriculum. To support dissemination and implementation of the curriculum across a wider range of universities, a lecturer peer support network - RASES[33] has been created in partnership with The Physiological Society. This scheme involves ‘educator experts’ from U.K. universities who are actively promoting the curriculum, supporting the development of educational resources[34] and engaging with academics from a range of Higher Education Institutions providing support where needed. We have also had international interest with a recent article[35] appearing in the August 2021 Pharmacology Matters.

3.4             19 universities expressed interest in accessing RASES support and 9 of these received support in its first year. Importantly this scheme will evolve to support the needs of the educator community. A key aspect of the RASES scheme is the identification and curation of educational e-learning resources that can supplement student teaching and learning.

3.5             Learned societies have important, direct relationships with researchers and educators, and are therefore well-placed to drive and support ambitions. UKRI should explore opportunities to collaborate with learned societies to leverage such impact. For example, RASES. Focusing resource on educational interventions can help improve research reproducibility for the long-term.

(September 2021)