Written Evidence Submitted by Dr Simon Kolstoe

(C190086)

Dr Simon Kolstoe

Senior Lecturer in Evidence Based Healthcare, University of Portsmouth*, UK

Independent Chair*, Public Health England Research Ethics and Governance Group 

Independent Chair*, Hampshire A (NHS) and Ministry of Defence Research Ethics Committees (MODREC)

*The evidence submitted here is not on behalf of the University of Portsmouth. As the independent ethics committee chair the opinions expressed here do not represent the views of PHE, NHS, HRA or the MOD.

Research Ethics in COVID-19: long vs short science

Summary

1. Since January 2020 Research Ethics Committees have rapidly reviewed COVID-19 related protocols, often in a matter of hours.

2. From experience as an independent research ethics committee chair, approximately half of the COVID-19 related protocols are well justified, designed and likely to produce robust evidence. The other half are hastily put together and with little scientific justification, often as a reaction to short term, poorly articulated, requests.

3. It has been estimated that even in normal times “85% of research is wasted, usually because it asks the wrong questions, is badly designed, not published or poorly reported.”[1]

4. The public and politicians are not well equipped to judge the quality or reliability of scientific research. An inability to distinguish between good and bad research leads to poor decision making.

5. Research Ethics Committees (as already constituted) could play an expanded role in helping to judge the quality of research evidence. Their reviews are fundamentally different from the scientific advice currently received by government.

It’s hard to think of another time when science has not just been front page news but has completely dominated the attention of the world’s media and public.

British Science Association (BSA)[2]

1             According to many measures[3] the UK is a world leader in research. This places the UK in an advantageous position for addressing the COVID-19 health crisis. Our research infrastructure has rapidly re-focussed onto COVID-19 meaning that results, in terms of both publications[4] and clinical innovations[5], are now forthcoming.

2             The mechanism for re-focussing research has come through explicit changes in funding priorities[6] with the desired knock-on effect of causing scientists, researchers and research establishments to divert their efforts to focus on the new priorities. But, while changed funding targets can cause rapid changes in research activities, robust and reliable research itself cannot be rushed.

3             This is not to say that increased funding cannot speed up research, but rather that such acceleration is caused solely by the increase in availability of resources, reagents and staff. For instance the RECOVERY trial that identified the benefit of low dose dexamethasone[7] was based upon a well-used “pragmatic trial design[8]” using existing drugs, while even the Oxford vaccines trial that has shown promising preliminary results[9] is using a modified form of an existing vaccine.

4             It is therefore important to appreciate that for the majority of research questions, increased funding cannot necessarily shorten the time needed to design the research, gather results, analyse evidence or publish results in a carefully scrutinised manner[10]. Time is of equal importance to money when it comes to producing robust, reliable and trustworthy research.

Research Waste

5             Chalmers and Glaziou have argued that even in normal times:

85% of research is wasted, usually because it asks the wrong questions, is badly designed, not published or poorly reported.[11]

6             In a subsequent series of papers edited by the same authors in the Lancet entitled “Research: increasing value, reducing waste”, five key reasons for research waste were identified: poor design of research questions, choice of inappropriate methods, inefficient regulation, inaccessible (often unpublished) results and biased research reports (figure 1).[12]

7             Slow progress has been made in addressing the problem of research waste, although the 2018 “Research Integrity” report from the House of Commons Science and Technology Committee[13] galvanised a certain amount of action including reports on Research Culture from the Wellcome Trust[14], research incentives from UKRI[15], a transparency initiative from the Health Research Authority[16] and a new version of the “Concordat to Support Research Integrity[17]” from Universities UK in collaboration with the UK Research Integrity Office (UKRIO). This latter concordat was endorsed by the Government’s Chief Scientific Advisor Sir Patrick Vallance[18]. In addition UKRI committed to establish a Research Integrity Committee by Summer 2020[19].

8             While this activity is encouraging, there is still a long way to go before the concerns raised by Chalmers and Glasziou, and echoed by the Commons Science and Technology Select Committee, will be fully addressed.

Long vs Short Science

9             The rapid acceleration of research caused by COVID-19 has likely increased the problem of research waste. While fraudulent research leading to retractions are generally fairly rare (such as the infamous case of MMR and Andrew Wakefield[20]) it is important to recognise that even in non-pandemic times research waste is a problem leading to the waste of a huge amount of research funding (often estimated in the $100 billions/year)[21].

10         During the last few months of COVID-19 we have seen funders rapidly implement new research calls[22], expedited review processes[23], methodologies hurriedly constructed[24], research timelines compressed[25], and publications rushed through[26]. If research waste is a problem normally, the rapid activity focussing on COVID-19 is likely to have magnified the problem[27].

11         Parallels with the 2009 Swine Flu pandemic can be drawn where the House of Commons Committee of Public Accounts found that the Department of Health wasted £424 million stockpiling Tamiflu, a drug that was found to have little effect[28]. This was due to rushed decision making based upon incomplete, and under reported research evidence.

12         It is therefore critical to appreciate that scientific research is a complex, iterative, and long process that often takes many years to reach a consensus. But the beneficial results of “long science” are indisputable in the form of all the medical, engineering and technological innovations that underpin modern societies.

13         Unfortunately the pressures of COVID-19 mean that the research community are being expected to reach a consensus on a variety of important issues extremely rapidly. This represents “short science”.

14         Although the content of proposals reviewed by research ethics committees must be kept confidential, the experience of chairing research ethics committees within the NHS, PHE, MOD and University, seem to support this assessment. Even in normal times researchers do not always do a good job of justifying their research proposals within ethics applications, but since January this year we have increasingly had to return proposals to researchers stating that they need a better rationale for their study rather than just saying “…because COVID-19”. We have previously reviewed the aspects of proposals that RECs commonly consider[29], and have anecdotally noticed that while remaining robust in terms of arrangements for protecting human participants, COVID-19 applications do tend to spend less time on the scientific justification of their research questions, emphasising short over long science.

15         More widely, although it must be acknowledged that the political system does play an important role in focussing attention on relevant social issues that need rapid answers, it should also be noted that the “Haldane Principle” was included in the Higher Education and Research Act 2017[30] which enshrined in law the independence of research, and the importance of robust peer assessment prior to awarding research grants. Of course during a time of emergency it makes sense to relax some principles, but at the same time it should be acknowledged that principles exist for a reason.

16         The reasons for an increased amount of short science in response to COVID-19 is understandable, and in one respect could be considered commendable as it represents a willingness by researchers to address the problem at hand. However, expecting short science to produce the same benefits as long science is unrealistic. When pushed, researchers will of course provide an opinion based on their expertise and current understanding of the evidence. But, while such opinions are probably more reliable than opinion from non-experts, they do not represent the robust scientific evidence created from long science and a stricter interpretation of the Haldane principle. Even a considered guess is still a guess.

17         Furthermore, education and training in the research/scientific method accustoms researchers to creating a hypothesis, testing the hypothesis, and then if required modifying their hypothesis in light of new evidence. As a result researchers and scientists, perhaps more so than other parts of the population, are more likely to change their minds as the evidence base evolves. Whereas contradiction is seen as a weakness for a politician or many other members of society, it is often seen as a strength for a researchers/scientists because it emphasises the critical importance of having an open mind.

18         Unfortunately this characteristic of science/scientists (being willing to change their minds based on new evidence) is not well understood more generally, meaning that the inevitable contradictions caused by the marketing of COVID-19 related short science, risks undermining the important, reliable, conclusions created by long science.

A solution – expanding the role of Research Ethics Committees (RECs)

19         The brief introduction presented above argues that the emergency situation caused by COVID-19 has caused an increased emphasis on short science, and increased potential for research waste. This is because short science is not as reliable as long science, and the tentative (and often wrong or evolving) conclusions of short science risks obscuring the robust, reliable and trustworthy evidence created by longer term science. Furthermore short science can also form the basis of poor decision making by those in authority.

20         And yet, especially in a time of public emergency, it is vital to still use evidence based decision making. One solution would be to make better use of Research Ethics Committees (RECs).

21         Research Ethics Committees exist to:

..have a dual mission to protect the rights, safety, dignity and well-being of research participants and to facilitate and promote ethical research that is of potential benefit to participants, science and society. [31]

22         Historically, and based upon the Declaration of Helsinki and other similar documents[32], RECs have primarily focussed on the care of research participants. But, although this role is still vital, over recent years the importance of also supporting researchers to conduct the best research has been increasingly emphasised[33]. Bad science is bad ethics because it wastes participants time along with research resources. This contributes to research waste, so is something that RECs increasingly look out for.

23         RECs in the UK are either i) recognised by the UK Ethics Committee Authority (UKECA) to review studies under specific laws such as the Clinical Trials Regulations, ii) hosted by a specific appointing authority for NHS research (Health Research Authority in England, Health and Social Care in Northern Ireland, the Chief Scientists Office in Scotland, or Health and Care Research Wales), or iii) are constituted by Universities and other research sponsors for non NHS research. In England committees overseen by the Health Research Authority (including UKECA recognised committees) all review according to established standards[34], while RECs overseen by other organisations vary widely in the robustness of their review processes, although attempts are being made to standardise reviews[35].

24         Taking HRA/UKECA recognised committees as the current standard for review, committees are made up of seven or more “independent” members who must include at least one expert member, and one member with no experience of health or social care research. In considering an experimental protocol RECs consider the following questions:

• Is there a description, unambiguous research question, and purpose?
• Is the study built on what is known already?
• Will the study provide meaningful answers to the research question?
• Will the study provide valid answers to the research question?
• Does the research team have the experience, skills, facilities and time to complete the study?
• Has the research incorporated patient and participant views?
• Are participants recruited with justifiable inclusion and exclusion criteria?
• Is there a fair balance of benefits and harms (risks) for all with an interest in the study?
• Is personal data handled appropriately (confidentiality)?
• Have participants been offered a fair choice through the information they are given (presented in plain English) and consent process?
• Are there fair payments for participation and financial recompense in case of harm?
• Will participants receive appropriate care both during and after the study?
• Do participants have access to an independent complaints procedure (or advocate)?
• How will results be analysed and disseminated?

25         These questions emphasise the dual role of a REC in both protecting participants AND considering the potential benefit to participants, science and society. They are also important questions that are relevant for examining any piece of research.

26         However, as currently constituted REC review occurs at a point in time BEFORE the research commences. Although amendments to protocols and safety issues are considered as studies progress, the responsibility for subsequent study evaluation sits with research sponsors not the research ethics committees themselves. Indeed RECs are often powerless to regulate research and researchers after studies commence[36].

27         Crucially, when results are reported or published, RECs play no role. Instead consideration of results is handled by journals who often use no more than two peer reviewers. Often peer reviewers are chosen on the basis of availability rather than specific expertise or competence[37]. Furthermore the increased used of “pre-prints” (the publication of draft scientific manuscripts on special pre-print servers[38]) mean that increasingly (and especially during COVID-19[39]) research is reaching the public domain prior to any quality check whatsoever.

28         If we already have RECs constituted especially to review research before it begins, why not also utilise them to review the results of research? Given the large number of scientific papers published they would not be able to review all research, but they could be used within government to provide an independent opinion specifically regarding any exceptional research that might be critical for decision making.

29         Of course government already has various scientific and other advisory committees[40], but critically such committees are appointed based upon specific expertise, often made up of eminent members of the scientific, engineering or medical communities. Research ethics committees are constituted very differently[41]. Although they do contain medical and scientific experts, such experts are often more junior and thus closer to the lab bench or bedside compared to many government advisors who often represent the senior echelons of their professions. Additionally experts make up the minority of REC members, sitting alongside “lay” members who represent a wide range of other communities. This makes a big difference because reviewers tend to identify issues in areas that they already have expertise in[42]. As a result “expert” scientific advice will always be biased to a certain extent[43], whereas research ethics committees are deliberately constituted in a way so as to minimise as much bias as possible. Their priority is also the patient and public perspective, rather than lobbying on behalf of the scientific or medical communities[44].

30         Having an independent REC review the evidence that is being used for important decisions will not prevent the use of bad (or unreliable) evidence, but would provide an additional and crucially independent process that differs from current government scientific advice. Additional reviews by RECs would help to promote both transparency and accountability of evidence based decision making.

31         This proposal is not to replace current expert committees that provide a critical role in both the setting of research policy/direction and subsequently advising government, but rather adding the potential for an independent, ethics focussed, review of critical evidence that may underpin important decisions.

32         It would also not add much time to any decision making processes. COVID-19 has demonstrated that established RECs can review important projects within hours. A carefully constituted, and resourced, REC would be well able to review research and help Ministers or others distinguish between short and long science when making crucial decisions in the public interest.

(July 2020)