Written evidence submitted by The Animals in Science Committee (RIN0048)
The Animals in Science Committee (ASC) is an advisory non-departmental public body of the Home Office. The Chair of the Animals in Science Committee is Dr John Landers and there are twelve members of the Committee, whose names can be found at:
https://www.gov.uk/government/organisations/animals-in-science-committee/about/membership
The Animals in Science Committee was established by the Animals (Scientific Procedures) Act 1986 as amended to comply with Directive EU 2010/63/EU which came in to force on the 1st January 2013. Article 49 of this Directive requires each EU country to set up a National Committee for the Protection of Animals used for Scientific Purposes. In this country the Committee is known as the Animals in Science Committee and has superseded the Animal Procedures Committee.
The roles of the Animals in Science Committee are:
The Animals in Science Committee is responsible for providing impartial, balanced and objective advice to the Secretary of State, to animal welfare bodies and within the European Union on issues relating to the Animals (Scientific Procedures) Act 1986 as amended.
SUMMARY
The ASC would support:
The ASC would not support:
Overview
The ASC agrees that the burgeoning incidence of research misconduct makes it necessary for procedures to be put in place that aim to 1) prevent research misconduct and 2) allow the implementation of appropriate sanctions. However, the resource implications of such measures will be considerable. There would also need to be clarification of the difference between misconduct and negligence.
From the perspective of the ASC, research integrity includes the need to ensure that research, which uses animals, is ethical and maximizes the benefit : harm ratio. Assurance on this point demands attention to all levels of the research process: individual experiments, the project, the institution and, sometimes, societal factors. We note that POSTNOTE attends primarily to the integrity of researchers and their research outputs.
1.1 Policies, Guidelines and Codes of Conduct
We agree with POSTNOTE’s inference that cases of deliberate misconduct are extremely rare and that questionable research practices (e.g. flawed data management or experimental design) are typically unintentional. However, we acknowledge that all forms of poor practice harm the research record and that it would be helpful to develop processes that strengthen the quality and integrity of (and public confidence in) research outputs.
To that end, we agree that all research establishments need a robust system for investigating allegations of research misconduct. Disclosure of the outcomes should be transparent, not least because some allegations of misconduct are lodged with malicious intent. Transparent exoneration of allegations of research misconduct is as important as their affirmation.
It is not clear how making research funding conditional on disclosure would work, in practice. Whereas individual academics, who are guilty of misconduct, could be denied prospective funding, it would be problematic to apply this sanction at the institutional level.
1.2 The Role of Peer Review
The peer review process is under immense pressure. This is a result of many factors, including the increase in research outputs, internationally, and the increasing reluctance of scientists to contribute to the peer-review process on the basis that it intrudes on research time. As a consequence, a good deal of peer-reviewing is carried out by early-career researchers because it enhances their professional portfolio. These researchers may be thorough and fair, but there is scope for strengthening these and other imperatives through systematic training in the peer review process.
Also, early career researchers are rarely the most experienced expert in the field and it is experience, which has accumulated over many years of specialist research that is often essential to spot data mismanagement, flawed interpretation of the results, or falsification. This limitation becomes even more prominent for research outputs that incorporate a range of research disciplines and technologies and/or large datasets.
Some initiatives are attempting to resolve these problems by inviting scientists to log their peer review assignments on a database, thereby enabling reviewers to gain credit for their efforts (e.g., Publons). However, the Research Excellence Framework (REF) does not consider these metrics and so many scientists see such facilities as unnecessarily bureaucratic and do not subscribe. Nevertheless, such a process, or something similar, is badly needed to increase the motivation of all scientists, particularly senior ones, to contribute to the peer review process.
In addition, Journal Editors may not be best equipped when it comes to selecting the most appropriate reviewers. The system relies on individuals opting to decline a request to review a manuscript when they are not ideally qualified to fulfil that role. Databases, such as Publons, could offer a better way of gaining insight into the expertise / experience of reviewers, but such tools are not yet used routinely by Editors (or Higher Education Institutions (HEIs)).
As acknowledged in POSTNOTE, another limitation of the peer review process is that it scrutinizes the description of the methods and interpretation of the results but rarely, if ever, the raw data. Whereas it would not be feasible (or cost – effective) to peer review all raw data, it might be helpful, if only as a deterrent, to develop a system for random auditing of individual data‑sets (“pour encourager les autres”; see also: Section 3.7).
The peer review process is typically unpaid. We would not support the introduction of payment for such work because it would be unlikely to increase the scrutiny of the outputs. Indeed, scrutiny could suffer if reviewers were tempted to overtrade because they regarded this work as a significant, additional source of income. Also, the costs would inevitably be underwritten by the organizations paying for the research and so diminish funds for the research it is intended to scrutinize.
We believe that informal peer review (e.g., pre-publication presentations at conferences) can be helpful because it enables scrutiny of the research at an early stage of the process, which can prevent misconduct (either inadvertent or deliberate), later. Many learned societies do operate peer review of these outputs (e.g., proceedings of their meetings), whereas large international congresses do not. However, it would be difficult, if not impossible, to require pre-publication peer review in the latter situation. In our view, the risk that publication of such informal outputs could be misinterpreted as a kite-mark of the quality of the output is minimal. The research community is generally aware of the status of these different types of output.
1.3 Sanctions
In our view, the majority of researchers strive hard to produce high quality research. The stigma associated with the retraction of a research output and, for academics, possible denial of future research funding are effective deterrents, in most cases. However, the penalties for committing a misdemeanour should be applied consistently, which is not the case at present.
Research that involves the use of animals can involve types of misconduct that were not considered in POSTNOTE: specifically, the mistreatment or unethical use of experimental animals. This is one field of research where there are clear legal sanctions whenever there has been any breach under the Animals (Scientific Procedures) Act. All reports of non-compliance, even self-reported ones, are investigated by the (Home Office) Animals in Science Regulation Unit (ASRU). Depending on the gravity of the misconduct, sanctions can range from a letter of admonishment from the Secretary of State to revocation of the license to carry out any further research using animals. For the most serious cases, there is provision for individuals responsible for the misconduct to be prosecuted in the criminal courts.
One of the reasons for legal sanctions in animal research is to maintain public confidence that any wrongdoing will be punished. We believe that the public confidence in scientific research would similarly be boosted if there were sanctions against deliberate research misconduct. It follows that prosecution would be appropriate, or should even be expected, for any type of misconduct that meets the threshold for a criminal act under UK law (e.g., fraud).
We acknowledge that criminalization of serious research misconduct will confront problems highlighted in POSTNOTE, viz:
We also agree that criminalization places the emphasis on individual researchers, and fails to have sufficient regard for wider, systemic problems in research, which contribute to misconduct.
2.1 Demonstrating Research Impact
The primary motivation for a research career has shifted progressively from a vocational activity to one that offers financial gain. For organizations that host research, the gains range from grant income to sales of products of the research. For individuals, the gains follow directly (e.g., royalties from IP) and/or indirectly (e.g., promotion).
This motivational shift has been aggravated by Research Assessment Exercise (RAE) and REF, which have imposed immense pressure to produce research outputs with high impact (either in terms of journal profile or for society, generally). Against this background, it is not surprising that the incidence of retractions of scientific publications is increasing and is highest for journals with a high impact factor.
2.2 The Changing Nature of Research and its Practice
It has become established practice that researchers exaggerate the expected benefits from their research (“…X...will lead to a new treatment for…Y.”). This is at least partly to help increase the likelihood of attracting further research funding in a highly competitive environment. Nevertheless, researchers should be encouraged to be more circumspect about the significance of their published and prospective research findings. Specifically, scientists should be precise about the extent to which their findings match the objectives of the study and offer realistic assessments of how they have enhanced progress in the field. This is not merely a matter of scientific integrity. For research that uses animals, is essential for assessing the harms to animals versus the benefit(s) that emerge from the research which, in turn, is needed to address societal concerns about the justification for animal experiments.
2.3 Funding Sources
An obligation to disclose the source of research funding has been adopted by many organizations and journals and there is scope for making these declarations mandatory. However, it is not clear whether this would make a difference to research integrity. On the one hand, disclosure signals transparency and the target audience might see this as assuring the reliability of the data. On the other hand, such disclosures can attract skepticism about the research findings, which could be unjustified. There is also a risk that disclosure of funding from a reputable funding body could induce a positive bias among reviewers/editors/readers. In the end, the reliability of the findings rests on the integrity of the researcher, regardless of any disclosure of who funded the work.
2.4 Three problems listed:
Academics rely on research outputs for career progression. It is unlikely that funding would be sought from any organization that blocks publication.
However, there is increasing concern about the reluctance of scientists, in any sector, to submit ‘negative’ results for publication, regardless of their intrinsic value. This is partly explained by journals’ reluctance to publish them, which culminates in unintentional repetition of such studies. In the case of animal studies, this has ethical implications because it causes unnecessary use of animals in experiments. For this, and other reasons, we strongly encourage publication of all research findings. To that end, there is a need to persuade journals to publish reliable hypothesis-driven data, even if the findings did not pan out as predicted.
We further suggest that the prejudicial description of findings as ‘negative’ should be strongly discouraged. Typically, negative findings are those that do not fit the hypothesis, or which do not replicate published work. Instead of perceiving the experiment to have ‘failed’, we believe it is merely a matter of reporting such findings in a positive way: i.e., in terms of what has been learned from the study. Such a change in the manner of reporting research would make it unnecessary to relegate such data to a separate archive of ‘negative’ findings.
A statement about the research objectives and whether or not they were met is normally required for a study to qualify for publication, but this requirement is often not upheld. Assessment of claims that the objectives were / were not met, or whether alternative explanations have been ignored, should be regarded as an essential component of the peer review process.
However, interpretation of data is often not clear-cut. This is sometimes (but not invariably) because the experimental design was flawed, which is a reflection of the lack of appropriate expertise in many research establishments. Whereas commercial laboratories often have access to experts in study design and data analysis, whose role is to advise the researchers, this is not the case for academic research centres. In our view, there is a pressing need for HEIs to prioritize the creation of academic posts to fill this vacuum in expertise in experimental design, data management and statistical analysis.
Another scenario is that the stated objectives of the study are ‘reconfigured’, post hoc: i.e., after the data have been gathered. This manoeuvre, which could be regarded as misconduct, would be prevented if scientists were required to register / archive both their research objectives and the study design, before starting the work. Such a process would enable subsequent assessment of whether any changes had been incorporated after the study started and, if so, whether they were justified.
For clinical research, some journals insist that every study design is peer-reviewed and archived, prospectively: these processes are a precondition for publication of the completed work. A similar process has been proposed for the preclinical biomedical sciences. However, we note that funding for academic research is typically approved after detailed scrutiny and peer-review of the study’s aims and design. In the case of all research using animals, the project has also been approved by an Animal Welfare and Ethical Review Body (AWERB) as an absolute requirement for compliance with A(SP)A.
In view of this background, we believe that it is not necessary (or feasible) for every preclinical study to require a further tier of peer-approval before the work starts. Instead, we believe that archiving of the study design and data (e.g., using Figshare) and the data analysis; would meet the intended purpose.
We agree that failure to publish certain types of findings, through commercial sensitivity or because they did not confirm the preferred hypothesis, can distort the scientific record. This is another reason why publication of all research findings, whether they met the research objectives or not, should be strongly encouraged (see also: Section 2.2). However, to avoid any publication bias that can distort meta-analyses, this obligation would need to be met internationally, not just within the UK.
3.1 Improving Openness and Transparency
We support Open Access publication, but it is not obvious how this would enhance research integrity.
We fully endorse the decision to make OA publishing mandatory for all publications reporting research that was funded by RCUK and wealthy charities (e.g., Wellcome Trust). However, we are uneasy about Open Access being mandatory, for all research publications emanating from the UK. This is entirely because small charities, which make a vital contribution to the research portfolio, do not have the resources to fund Open Access publishing.
Nevertheless, we acknowledge that Open Access can only help to reduce unnecessary duplication of research. In the case of research using animals that would lead to a welcome reduction in the numbers of animals subjected to scientific experiments, worldwide. To that end, we would fully support the development of a funding mechanism to enable all UK research outputs (including research funded by small charities) to meet this aspiration.
3.2 Publishing data-sets:
We support the principle of data-sharing, as promoted in the Concordat on Open Research Data, and respect the intentions of journals that insist on data-sharing as a condition for publication. However, we believe there is a distinction between the sharing of archived data (i.e., requires the consent of the researcher) and open access to archived data (no consent required). Whereas we would support making the former a condition of funding, we would support the latter only on a voluntary basis.
3.3 Checklists:
We strongly support the requirement for checklists (as a qualification for publication), which should include details of the statistical analysis, sample sizes, computer codes and so on.
We further propose that, for research that has used animals, compliance with the ARRIVE guidelines (or their equivalent) is mandatory. Some leading journals have already adopted this guidance and others have devised versions that are more suitable for their field of research (McGrath et al., 2016). Nevertheless, even two years after the publication of the ARRIVE guidelines there had been no significant improvement in the quality of reporting of animal studies in top-tier journals (i.e. PLoS and Nature publishing groups; Baker et al., 2014), suggesting that peer reviewers and editors, not just authors/scientists, need to have greater awareness of this aspect of research integrity.
Findings from two, ongoing projects that aim to establish the utility of checklists will be helpful in this respect. One of these is a ‘before and after’ study being carried out by the journal, Nature [https://link.springer.com/article/10.1007/s11192-016-1964-8]. The other is a controlled trial of mandatory checklist compliance, which randomised 1600 submissions to a leading journal [see: https://ecrf1.clinicaltrials.ed.ac.uk/iicarus]. Both studies are due to report in September 2017.
We further note that, over the past decade, some journals require reviewers to disclose their identity. There are obvious benefits and drawbacks to this initiative and it is not yet clear whether, or how, this has affected the rigour of the review process. So far, evidence suggests that open reviews are more likely to recommend acceptance [http://bjp.rcpsych.org/content/176/1/47]. However, there appears to be no substantial difference in quality of the review, the recommendation regarding publication, or the time taken to review, but there was a significant increased the likelihood of reviewers declining to review [https://www.ncbi.nlm.nih.gov/pmc/articles/PMC27670/].
3.4 Reporting on clinical trials:
No comment
3.5 Oversight, Training and Sharing Best Practice
For research using animals, there is mandatory oversight of research conduct by the Animals in Science Regulatory Unit (ASRU) but this does not extend to scrutiny of the data output.
In other respects, effective research teams must operate on a foundation of trust, without which they cannot function in any meaningful way. In our view, it is more important to ensure that there is a robust climate of integrity, alongside a realistic level of scrutiny by the PI and colleagues. We agree that effective oversight of the raw data by the PI is desirable, but this is not always feasible, except on a random sampling basis.
We acknowledge the risk that scientists/technicians could modify their behaviour (renounce behaving with optimal integrity) due to peer-pressure. For instance, a junior scientist may be reluctant to report or challenge questionable research practice from a senior scientist; technicians may be tempted to ‘cut corners’ out of time pressure, or to avoid filling complex, and potentially compromising reporting forms. While most research institutions have now incorporated some form of performance management for all staff (i.e., PDRs), few offer an anonymous specialized reporting system. However, routes for 'raising concerns' (often anonymously) are becoming more common in institutions and enable people to disclose animal welfare issues (or other forms of poor scientific practice). One that could serve as a model for matters involving research integrity, more generally, is a specialized system that encourages animal technicians to report any matters of professional concern [http://www.agendalifesciences.co.uk/welfare-first].
Another approach could be a system for anonymous 360-degree feedback, which would help to identify technicians and scientists who tend to ‘cut corners’ or those who put pressure on others to act sub-optimally. However, people's inclination to utilise these avenues are likely to be strongly affected by whether the institution has a good culture of care, which encourages people to speak up (without fear of repercussion) and with confidence that their concerns will be acted on.
In respect of training and CPD, we believe that all such courses should include elements of scientific integrity and practice. For research involving animals, this could be incorporated into the training that must be completed before applying for a personal licence (PIL) to carry out the work. Such training (i.e., Animal (Scientific Procedures) Act [A(SP)A} / EU Directive modules) is compulsory from the moment a scientist joins a UK research platform. So far, research integrity is not clearly spelled out in this training (or is inconsistently taught). In the case of more senior scientists (PIs) who are applying for a project licence (PPL), this process could focus on their additional obligations, such as proper oversight of everyone working under their supervision.
In short, we would support the development of a compulsory research integrity curriculum for scientists and technicians, working in the UK. This training should cover topics such as: data collection (proper use of the lab book), study design, stats, etc. and should be incorporated into the A(SP)A compulsory training modules. The ‘named person’ within A(SP)A who deals with training (the Named Training and Competence Officer [NTCO]) would be responsible for sourcing appropriate courses/trainers for these subject areas. Making both scientists and the general public aware of the societal impact of bad science (e.g., the reference to the MMR debacle in POSTNOTE) would help to convince scientists of the need for such courses.
Finally, in this context, we would support greater collaboration and dialogue, not only between university staff (such as the Russell Group Research Integrity Forum) but also academics, funders, publishers and UKRIO.
3.6 Realigning incentives for Researchers
We would support the inclusion of a research integrity statement in REF (2021), which would motivate universities and researchers to go beyond the ‘minimum’ to improve practices. This would be facilitated by proposals to decouple researchers from research outputs and so place greater emphasis on the ‘research environment’, rather the behaviour of individual scientists.
We would support the broadening of the definition of research impact to include, for instance, whether their work has been replicated (see: Section 3.7). We would also hope that REF would regard impact as including an advance in one or more of the 3Rs (Replacement / Reduction / Refinement) that underpin laboratory animal science. Examples could include the development of a technique that reduces the number of animals needed to meet a research objective, or which improves the welfare of experimental animals, or replaces the use of animals. Such progress helps to resolve societal concerns about the use of animals in scientific research. Yet, hitherto, the ‘impact’ of such work has not been rated highly by RAE / REF.
Furthermore, we believe that all publications should include a section specifying the contribution of all authors. On that basis, there would be transparency in respect of the individual(s) who are directly responsible for the cause of the retraction / mistake. It would also ensure that senior researchers cannot consolidate their reputation by ‘piggy-backing’ on the achievements of junior colleagues (claiming the credit for research of high impact), but shed responsibility for research of questionable integrity.
We agree that the stigma attached to retractions can deter authors from correcting their scientific record. However, in such cases, there is an implicit understanding of the difference between deliberate misconduct (e.g., fraud) and inadvertent mistakes.
The funding periods that have prevailed (typically 3 or 5 years) work well. These days, all applications incorporate clear milestones that have been judged as realistic by peer review and so there should not be undue pressure to produce the promised outputs. Extension of the funding period would lead to fewer projects being funded, overall, which would be particularly harmful for early-career scientists.
Research findings are continually subjected to independent challenge as an essential element of the scientific process. However, we believe that research using animals increases the imperative to produce research findings that can be replicated or, if not, to determine the reason behind the failure to replicate.
Successful replication is undoubtedly predicated on full reporting / disclosure (see: Section 3.3). Even then, it is possible that some unreported factor accounts for a failure to replicate. Recent experience confirms that tests of replicability require many such challenges, with some succeeding in replication, but others failing to do so. The most important objective is then to identify the key factor(s) that underlie successful or failed replication: it should not be assumed that flawed scientific practice is the only explanation.
To encourage journals to place more emphasis on publishing replications would fundamentally change the culture of publication, for which novelty is a key qualifying criterion. Although there are already facilities that offer a conduit for replication (e.g. PlosOne, Figshare), these are not highly regarded by REF. As a consequence, to expect journals to publish replications would require them to abandon the impact factor as an indication of the importance of the published work. For commercial reasons, this will be hard to achieve.
A research finding that does not support the preferred hypothesis is often described as a ‘negative’ finding. However, in our view, the only negative results are ones where the authors cannot be certain about the reliability of their findings because the experiment was flawed through its design and / or execution. To change the profile of a ‘negative’ result (i.e., one that did not support the hypothesis) into a positive finding (i.e., the finding points to a new line of research) is merely a matter of journalism. Arising from that, we believe that the establishment of a repository specifically for ‘negative’ data could seriously undermine the potential importance of such data (see also, Section 2.4). Instead, data that are not accepted for publication on the grounds that they are not novel, or interesting enough, could be reposited in any of the standard databases.
3.7 A Regulatory Body in the UK?
We do not support the establishment of a regulatory body, to oversee publically-funded research, beyond that already provided by the Animals in Science Regulation Unit (ASRU) (for research using animals). This is not least because this could lead to skepticism about research funded by less wealthy organizations, that does not receive such scrutiny. Also, prevention is better than prosecution.
Also, we agree that:
However, there is an argument for a body charged with leading improvements in biomedical research in the UK, regardless of the funder, and with a focus on “audit for improvement” rather than “audit for accountability”. We see this as particularly appropriate for research using animals because such research carries an additional ethical obligation to ensure replicability.
One option is to establish a body that tests the replicability of randomly selected samples of data (including ‘negative’ results), which would provide the reassurance that is essential for data-sharing. Such a body would have to be fluid, with the process shared across different laboratories, so as to cover the necessary expertise.
Another possibility is for independent replication of key finding(s) to be commissioned by researchers, which would provide a kite-mark for the work. Successful replication could be included in the research output / publication and the scientists who carried out the replication could share authorship of the work (Mogil & MacLeod, 2017).
However, we acknowledge that either of these processes would need considerable financial resource and the question of who would provide that would need to be resolved.
March 2017
References
Baker D, Lidster K, Sottomayor A, Amor S (2014). Two Years Later: Journals Are Not Yet Enforcing the ARRIVE Guidelines on Reporting Standards for Pre-Clinical Animal Studies J. A. Eisen, ed. PLoS Biology, 12(1): pp.e1001756–6.
McGrath JC, Lilley E (2015) Implementing guidelines on reporting research using animals (ARRIVE etc.): new requirements for publication in BJP. Br J Pharmacol. 172:3189-93. doi: 10.1111/bph.12955.
Mogil JS, Macleod MR (2017) No publication without confirmation. Nature. 542:409-411. doi: 10.1038/542409a.