Written evidence submitted by Dr Simon Kolstoe (RIN0022)

 

Ethics committees, managed though the Health Research Authority (HRA), are key for monitoring research integrity

 

Dr Simon Kolstoe, Independent Chair of Hampshire A NHS Research Ethics Committee; Independent Chair of MOD Research Ethics Committee; Senior Lecturer & Ethics Advisor University of Portsmouth 

Samantha Trace, Independent member, Hampshire B NHS Research Ethics Committee

Daniel Shanahan, publisher for Springer Nature's open access health sciences journals

 

This submission specifically addresses:

  1. The effectiveness of controls/regulation (formal and informal), and what further measures if any are needed;
  2. What matters should be for the research/academic community to deal with, and which for Government.

 

SUMMARY

 

s1.   There is no point conducting research if the results are not accurately communicated, but how research is communicated, and in how much detail, is not a straightforward issue.

 

s2.   On one hand, it must be accepted that those who fund research should be allowed to reap the rewards in advance of other companies or enterprises. However, as research projects using animals or humans are often burdensome, there is an ethical imperative to ensure that results are published within a reasonable time and with enough detail to avoid unnecessary replication. When this doesn’t happen the problem of reporting bias occurs.

 

s3.   Reporting bias is a research integrity issue, and has caused thousands of deaths within worldwide medicine, as medical doctors are not given access to the full information on the drugs they are prescribing[1],[2]. It is also the reason for an incredible waste of public funds such as during the H1N1 “Swine Flu” epidemic where £424 million was spent on Tamiflu by the British Government[3]. Here, a subsequent systematic review of the evidence behind the clinical efficacy of Tamiflu found that eight out of the ten trials that were used by the company to show the drug was useful in preventing complications such as pneumonia had never actually been peer reviewed or published. Instead clinicians, and in this case governments, were relying on a marketing spiel claiming successful trials of this drug rather than being able to consider the actual evidence of the drug efficacy for themselves[4].

 

s4.   It is encouraging that the problem of reporting bias is becoming increasingly prominent both publicly and politically. Significant steps such as the enforcement of clinical trial registration have recently been made, but this progress does not directly address the problem of reporting bias in the scientific literature. Recently calls have been made for sponsors and funders to audit their research and ensure that data is being suitably published[5]. Whilst this is undoubtedly important, we argue that research ethics committees are in a far better place to determine whether research is being published, and whether the published outcomes reflect the original research. 

 

s5.   In the UK it is a requirement for researchers intending to conduct many types of human research to submit their protocols for review by National Health Service research ethics committees. In England, there are sixty-eight research ethics committees overseen by the Health Research Authority who, as the national ethics regulator, have access to all the significant, and currently confidential, research protocols. The Health Research Authority is therefore in the best position to support research ethics committees in monitoring publications arising from projects, and determine whether publications accurately reflect the outcomes originally described in research protocols. We have previously described an auditing method that can achieve this[6], and now call on the Health Research Authority to provide a model for the international community of how publication and outcome reporting bias can be effectively detected and monitored using research ethics committee records.

 

s6.   Although this process should be managed by the HRA, additional funding for this role should be raised from industry and investors, as they will directly benefit from ensuring there are no “skeletons in the cupboard” that might affect their investments.


Background

b1    The issue of how scientists communicate their results is of increasing public and political interest. [1,2] In the most straightforward case, when research is publicly funded, it seems reasonable that the public receives a return by gaining access to newly generated knowledge. This applies to work funded through government research councils as well as work subsidised by public health services. [3-5] Similarly, scientific publications are a good way for medical or scientific charities to demonstrate that the donations they receive are making a meaningful contributing towards fighting targeted diseases or solving relevant problems.

 

b2    The issue of how industry communicates research is slightly less clear as companies need to be allowed to make a return on their investments in research and development. However, in the field of health and particularly pharmaceuticals, there seems to be a strong ethical argument that as much information –either positive or negative - about any new treatments are disclosed in a timely manner to aid effective scientific and clinical decision-making. [5] While it may be reasonable to delay publication for a limited time due to commercial reasons, all previous work must be published eventually to ensure future resources are not wasted and patients are not harmed. In addition, and perhaps relevant to both industry and public or charity research, the often altruistic contribution or even sacrifice made by research participants creates an additional ethical obligation for all results to be disseminated appropriately and in a timely manner. [6]

 

Communicating scientific research

b3    There are a number of different ways that the results of scientific research can be communicated. [7] Traditional professional avenues include conferences, workshops, textbooks and articles in peer-reviewed journals. Results can also be communicated through more regulated avenues such as clinical manuals, drug information leaflets, specialist training events, annual reports or even reports to shareholders. More recently the internet has provided further options through websites, blogs and social media.

 

b4    The main factor that distinguishes between these communication types is the target audience. In most cases, the people in the best position to apply and build upon new knowledge are specialists working in the area, so it can be argued that communication to these specialists must be prioritised in order for the main ethical obligations to be met. Almost universally, specialists use articles in peer-reviewed journals as these provide readers with a certain level of confidence that the work has been conducted appropriately. This is not to argue that other types of dissemination are not important, but rather that the most significant ethical arguments for the communication of new scientific and medical knowledge are best addressed using the established peer-reviewed scientific and medical literature. [8]

 

b5    However, despite these ethical pressures, it has long be known that not all scientific results make it to journals - a problem known as publication bias.[9,10] Furthermore, despite peer-reviewed papers being commonly considered as the “gold standard” of scientific communication, it has been well documented that even papers in such journals can severely misrepresent or distort actual findings.[11,12] This can take many forms including selective reporting of outcomes and methods (outcome reporting bias),[13,14,15] and hypothesising after the results are known (HARKing).[16] These issues have recently been emphasised through a number of high-profile scandals, including incomplete publication of data relating to the anti-depressant drug Paroxetine,[17] and the purchase of £424 million worth of Tamiflu by the U.K. government during the 2009 “swine flu epidemic” predicated on incomplete and misleading clinical data.[18,19] Such scandals have led to international calls for regulators to address such issues more appropriately including the recent redrafting of the influential “Declaration of Helsinki” by the world medical organisation, and David Cameron’s 2015 call for a G7 agreement on trial transparency.[20]  In the UK, new legislative attempts include the 2014 Care Act that established the Health Research Authority (HRA) as a non-departmental public body with a responsibility to promote transparency in research.[21]

 

MONITORING TRANSPARENCY

b6    The HRA is particularly well placed for this task as it provides the 68 research ethics committees (RECs) that are responsible for reviewing all human research conducted in the NHS in England. RECs have access to project protocols prior to the start of research, updates and substantial amendments as the research progresses, and receive notification once the research is complete, sometimes including preliminary results.[22] As the HRA holds the national archives of these records, they are in a particularly strong position to determine whether researchers publish their results in the scientific literature (in order to detect publication bias), and whether the published outcomes are consistent with the outcomes examined by the researcher (to detect outcome reporting bias and HARKing).

 

 

b7    In a pilot experiment (Figure 1), one of us analysed projects submitted to the Hampshire A REC to determine whether it was feasible to use these records to monitor publication rates. [23] Literature searches were conducted for 116 completed projects over a specified timeframe and determined a publication rate of 32%. Of the projects that were published, 57% showed inconsistencies with the outcomes originally declared in the ethics application. This information was determined effectively from the REC records, aided in particular by the use of electronic ethics submissions. This pilot work demonstrated that those with access to REC records are in a particularly powerful position to detect publication and reporting bias in contrast to similar attempts conducted by research funders or systematic review organisations who do not have immediate access to such a wide range of otherwise confidential protocols.  However, this task was time consuming and occupied a graduate student for most of a year despite the relatively limited number of studies being audited. Given that a typical REC reviews between forty and eighty studies per calendar year, extra resources would be needed if individual RECs or those managing them were to take on this role more comprehensively.

 

b8    We therefore argue that the HRA need to do more to support the ethics process by effectively monitor publication and outcome reporting bias using REC records. Our pilot work suggests a method that could be used to provide definitive publication data for all research conducted using NHS patients and resources that submit to RECs in England. Provisions for audit are already contained in the declaration signed by investigators when they submit their projects for ethical review, and a combination of the electronic submission system (IRAS) and internal HRA databases, ensure that this data is readily available to those with the appropriate permissions. [24]

 

b9    The question of funding for a number of posts to analyse this data would not necessarily need to be met directly by the HRA as almost all significant research funders, sponsors and even city investors have already indicated their interest in ensuring research transparency through signing up to the AllTrials campaign. The HRA could therefore approach these other bodies for a shared contribution towards the cost of running such an operation.

 

 

 

NEXT STEPS

b10         Perhaps the only significant question would be what to do once the results are in and sponsors/researchers who do not publish are identified. It may be hoped that if researchers know that this data is being gathered, higher levels of publication may result, but if publications are not forthcoming the HRA will be in an excellent position to contact researchers and find out why publications have not been forthcoming. They could then consider imposing relevant (and proportionate) sanctions or administrative measures to encourage appropriate levels of research transparency. We acknowledge that the answer to this latter question is not simple, as it may require broad changes to the research landscape, [25] but the problem of what to do next should not be confused with the decision to find out, now, whether researchers publish their results. At a time when the HRA is still a newly established non-departmental public body, and prior to any announcement as to how it plans to specifically promote publication transparency, we call on the HRA to act decisively and effectively to address the problem of reporting bias once and for all in the projects they play a role in overseeing.

 

Conclusion

b11         Even beyond biomedicine, a humanitarian argument can be made that scientific progress can only be achieved on the shoulders of prior knowledge. However, this cumulative model of scientific progress only works if the reports of previous experiments – the scientific literature – are reliable. While there is increasing attention given to these issues, [26] regulation is of no use unless there is a system in place to monitor the problem and enforce it. For instance, it has been a legal obligation to report the results of all clinical trials in the USA on ClinicalTrials.gov for nearly a decade, with the FDA entitled to issue fines of $10,000 for every day that results are not posted after the set deadline. But to date fewer than half the trials registered on ClinicalTrials.gov have posted results, and no fines have been levied, [27] partly because no one with sufficient authority seems to be monitoring the issue. The Health Research Authority has the opportunity to go one better: both determining who the culprits are, and then acting decisively to ensure that as much research as possible gets published for the benefit of all.

 

 

 

b12 KEY MESSAGES:

 

 

 

b13 CONTRIBUTORS:

 

 

 

March 2017

 


b14 REFERENCES:

 

  1. Hill D, Rapoport AI, Lehming RF, Bell RK. Changing U.S. output of scientific articles: 1988–2003. National Science Foundation, Division of Science Resources Statistics, 2007. NSF 07-320. https://wayback.archive-it.org/5902/20160210143710/http://www.nsf.gov/statistics/nsf07320/ (accessed 22 Feb 2017).

 

  1. Finch J. Report of the Working Group on Expanding Access to Published Research Findings (Finch Report). Research Information Network, 2012. https://www.acu.ac.uk/research-information-network/finch-report (accessed 22 Feb 2017).

 

  1. Eisen M. Research Bought, Then Paid For. The New York Times 2012. http://www.nytimes.com/2012/01/11/opinion/research-bought-then-paid-for.html?_r=0 (accessed 22 Feb 2017).

 

  1. Howard J. Who Gets to See Published Research? The Chronicle of Higher Education 2012. http://chronicle.com/article/Hot-Type-Who-Gets-to-See/130403/ (accessed 22 Feb 2017).

 

  1. Parker M. The ethics of open access publishing. BMC Med Ethics 2013;14:16. doi:10.1186/1472-6939-14-16

 

  1. World Medical Association. WMA Declaration of Helsinki - Ethical Principles for Medical Research Involving Human Subjects. Updated 2013. http://www.wma.net/en/30publications/10policies/b3/ (accessed 22 Feb 2017).

 

  1. Stone R, Jasny B. Scientific Discourse: Buckling at the Seams. Science 2013;342(6154): 56–7. doi: 10.1126/science.342.6154.56

 

  1. Harley D. Scholarly Communication: Cultural Contexts, Evolving Models. Science 2013;342(6154): 80–2. doi: 10.1126/science.1243622

 

  1. Sterling TD. Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa. Am Stat Assoc J 1959;54(285):30–4. doi:10.2307/2282137

 

  1. Simes RJ. Publication bias: the case for an international registry of clinical trials. J Clin Oncol 1986;4(10):1529–41.

 

  1. Altman DG. The scandal of poor medical research. BMJ 1994;308(6924):283–4. doi: 10.1136/bmj.308.6924.283

 

  1. Ioannidis JP. Why most published research findings are false. PLoS Med 2005;2(8):e124. doi:    10.1371/journal.pmed.0020124

 

  1. Chan AW, Hrobjartsson A, Haahr MT, et al. Empirical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols to published articles. JAMA 2004;291(20):2457–65. doi: 10.1001/jama.291.20.2457

 

  1. Dwan K, Altman DG, Clarke M, et al. Evidence for the selective reporting of analyses and discrepancies in clinical trials: a systematic review of cohort studies of clinical trials. PLoS Med 2014, 11:e1001666. doi: 10.1371/journal.pmed.1001666

 

  1. Goldacre B. Make journals report clinical trials properly. Nature 2016, 530:7.  doi:10.1038/530007a

 

  1. Kerr NL. HARKing: hypothesizing after the results are known. Pers Soc Psychol Rev 1998;2(3):196–217.

 

  1. Angell M. Drug Companies & Doctors: A Story of Corruption. New York Review of Books 2009;56 (1). http://www.nybooks.com/articles/archives/2009/jan/15/drug-companies-doctorsa-story-of-corruption/ (accessed 22 Dec 2017).

 

  1. Jefferson T, Doshi P. Multisystem failure: the story of anti-influenza drugs. BMJ 2014; 348:g2263. doi: 10.1136/bmj.g2263

 

  1. National Audit Office. Access to clinical trial information and the stockpiling of Tamiflu. 2013. https://www.nao.org.uk/report/access-to-clinical-trial-information-and-the-stockpiling-of-tamiflu-2/ (accessed 22 Feb 2017).

 

  1. UK Government. Prime Minister calls for 'wake-up to the threat from disease outbreak'. 2015. https://www.gov.uk/government/news/prime-minister-calls-for-wake-up-to-the-threat-from-disease-outbreak (accessed 22 Feb 2017).

 

  1. Care Act 2014. http://www.legislation.gov.uk/ukpga/2014/23/contents/enacted/data.htm (accessed 22 Feb 2017)

 

  1. Health Research Authority (HRA) Standard Operating Procedures. http://www.hra.nhs.uk/resources/research-legislation-and-governance/standard-operating-procedures/ (accessed 22 Feb 2017).

 

  1. Begum R, Kolstoe SE. Can UK NHS research ethics committees effectively monitor publication and outcome reporting bias? BMC Med Ethics 2015;16:51. doi: 10.1186/s12910-015-0042-8.

 

  1. Integrated Research Application System (IRAS). https://www.myresearchproject.org.uk (accessed 22 Feb 2017).

 

  1. Bassler D, Mueller KF, Briel M, et al. Bias in dissemination of clinical research findings: structured OPEN framework of what, who and why, based on literature review and expert consensus. BMJ Open 2016;6:e010024. doi:10.1136/bmjopen-2015-010024

 

  1. Jones CW, Keil LG, Holland WC, Caughey MC, Platts-Mills TF. Comparison of registered and published outcomes in randomized controlled trials: a systematic review. BMC Med 2015;13:282. 10.1186/s12916-015-0520-3

 

27. Anderson ML, Chiswell K, Peterson ED, et al. Compliance with results reporting at ClinicalTrials.gov. N Engl J Med 2015;372(11):1031–9. doi: 10.1056/NEJMsa1409364

 

 

[1] BMJ, “The true lorcainide story revealed”, 350:g7717, 2015

[2] TED, “What doctors don’t know about the drugs they prescribe”, 2012

[3] National audit Office, “Access to clinical trial information and the stockpiling of Tamiflu”, HC 125, 2013

[4] Cochrane Database Sys Rev, “Neuraminidase inhibitors for preventing and treating influenza in adults and children”, 4:CD008965, 2014

[5] Tompson AC, Petit-Zeman S, Goldacre B, et al Getting our house in order: an audit of the registration and publication of clinical trials supported by the National Institute for Health Research Oxford Biomedical Research Centre and the Musculoskeletal Biomedical Research Unit BMJ Open 2016;6:e009285. doi: 10.1136/bmjopen-2015-009285

[6] Begum R, Kolstoe SE. Can UK NHS research ethics committees effectively monitor publication and outcome reporting bias? BMC Med Ethics 2015;16:51. doi: 10.1186/s12910-015-0042-8.