Science and Technology Committee
Oral evidence: Research integrity, HC 350
Monday 4 December 2017
Ordered by the House of Commons to be published on 4 December 2017.
Members present: Norman Lamb (Chair); Vicky Ford; Bill Grant; Clive Lewis; Stephen Metcalfe; Martin Whitfield.
Questions 277 - 360
Witnesses
I: Dr Ivan Oransky, Co-Founder, Retraction Watch, and Distinguished Writer in Residence, New York University Arthur Carter Journalism Institute; and Professor C K Gunsalus, Director, US National Centre for Professional and Research Ethics. [This evidence was taken by video link]
II: Dr Ben Goldacre, DataLab, Department of Primary Care, University of Oxford; Dr Simon Kolstoe, Senior Lecturer and University Ethics Adviser, University of Portsmouth, and Independent Chair of Hampshire A (NHS) and the MOD research ethics committees; and Síle Lane, Head of International Campaigns and Policy, Sense about Science.
Written evidence from witnesses:
Witnesses: Dr Oransky and Professor Gunsalus.
Q277 Chair: Welcome, both of you. Perhaps you could introduce yourselves briefly, starting with Dr Oransky.
Dr Oransky: Certainly. I am glad to be with you, here in New York. I am Ivan Oransky. I am the co-founder of Retraction Watch and distinguished writer in residence in New York University’s Carter Journalism Institute.
Q278 Chair: Thank you very much indeed.
Professor Gunsalus: I am C K Gunsalus, director of the National Centre for Professional and Research Ethics at the University of Illinois, housed in the Co-ordinated Science Laboratory. We were initiated with a large NSF grant to create a national online ethics resource centre. We do ethical leadership development from high school through executive education, with the emphasis on institutional integrity. I am a professor emerita of business and have served on the faculties of the colleges of law, medicine and business at the University of Illinois. In 1995, I served on the Ryan commission on research integrity, and on the recently completed 2017 National Academies of Science, Engineering and Medicine fostering integrity in research panel.
Q279 Chair: It is very good to have both of you with us. There are five of us here in London. We will be asking questions of you both. Could I direct the first question to Dr Oransky? The Retraction Watch blog has, I understand, been running for seven years. You describe yourself as “a long-time observer of scientific misconduct.” What do you think has changed in relation to retractions over this period, both quantitatively and qualitatively?
Dr Oransky: Thank you for the opportunity. Adam Marcus and I co‑founded Retraction Watch about seven and a half years ago in 2010. We had a very incomplete understanding of retractions then. By we, I mean not just me and Adam, but generally, all of us in the scientific community, the journalism community and the regulatory community. It turned out that retractions had grown tenfold from 2000 to 2010, unbeknown to us. I could claim that we had known all this but we did not. Nature did a very nice job of demonstrating that a year later.
Since then, the number of retractions has grown—I don’t think exponentially, even in a literal sense, is too far off. In the year 2000, there were about 30 retractions. By 2016, we had created a database, with very generous funding from the MacArthur Foundation, the Arnold Foundation and the Helmsley Trust, which is now essentially complete. We estimate that it is 97% complete. We have close to 16,000 retractions in that database. To make a comparison, in 2000, there were 30 retractions, and about 1,300 in 2016. Even if you look at the rate of papers being published, which has more than doubled, there is still—
Q280 Chair: Globally, the numbers have more than doubled. Is that right?
Dr Oransky: That is correct. Globally. If I may, I will give you some sense of the UK figures. You may recall from our initial written testimony, which we submitted about a year ago, that when we looked at a leading database of retractions we found 147 retractions since 1990. Now that we have built our database, we can come up with a more robust figure that includes many other sources and goes back further than 1990. We found just shy of 400 retractions; there were 396 or 397 retractions, about half of which were due to misconduct. In the UK, there were about 400 retractions going back 40 years to 1977. The rate has doubled every several years. If you look at different periods of time, the rate has doubled. To put that in context, the rate of retractions has doubled every few years for every country, globally, so the UK is on a level playing field when it comes to retractions; it is also on a level playing field when it comes to recent retractions. Where we can tell what they are due to, we now think that about half of those are due to misconduct.
I would put a couple of large caveats around any retraction figures. One is that a large number of retractions may reflect the improved or better health of the scientific enterprise, because people are actually paying attention. We have no reason to think that fraud and misconduct did not exist before 1977; we just know that it was far rarer before 2000, although there is some evidence that it is on the increase, so the level of retractions may simply be a screening or attention effect.
The other caveat is that what we found, and always stress, is that denominators are very difficult. The UK has 400 retractions over that period; the US has far more than that. In the preliminary analysis, we tried to compare countries that cluster in terms of R&D spending; that is, retractions per billion dollars spent on research. I am fully aware that this is an imperfect metric, to say the least, but it starts to give you a sense of what the denominator is. It turns out that globally the figure is about 0.44. The number of retractions per billion dollars spent on research is about 0.44, so it remains quite a rare event.
In the UK, it looks like the number is somewhat higher than that—0.75. If you compare countries that are clustering in terms of the annual percentage spend of R&D budget by GDP, the UK comes out a bit higher than Denmark and the US; it is similar to Australia. I do not want to take up more time on details, but I would be happy to send you these figures and some of our findings. I stress that they are very small numbers, and they may represent more attention being paid to papers in countries that have higher rates, but, globally speaking, on a very rough cut, in the UK and the US retractions are increasing and retractions for misconduct are increasing, but that is consistent with what we see globally.
Q281 Chair: You think there is an increase in misconduct, not just better recording of the information.
Dr Oransky: It is clear that of the two main reasons to do with more retractions there is better reporting, screening and analysis—more eyeballs on those papers, and therefore you would expect to see more. The data and the evidence are quite clear.
On the question of whether or not there is more misconduct, some evidence is beginning to emerge. I cite, for example, a paper by Elisabeth Bik and colleagues that appeared about a year and a half ago. A pre-print version was published and I believe the reference is in our initial testimony, but I would be happy to send it again. Dr Bik looked at 20,000 different papers and found that from 2000 to 2014 the rate of inappropriate image manipulation—taking a figure that said one thing and making it look as though it said something a bit better in terms of the results—had doubled during that period. Because Photoshop is available, people can not only take the image but manipulate it, which stands to reason. I would not say it is clear yet, but there is some evidence that misconduct itself was on the rise until that point.
Q282 Chair: To be clear, the main countries are broadly comparable, with the UK having slightly higher rates of misconduct than some other jurisdictions.
Dr Oransky: I would not necessarily say it has higher rates of misconduct, but certainly a higher rate of retractions, half of which, to be fair, are due to misconduct. The denominators are different, but, on a rough-cut analysis, there is a slightly higher rate of retractions per money spent on research.
Q283 Chair: Do you think that the role you perform will always be needed, or are you addressing a phenomenon that can be sorted? There will always need to be vigilance, will there not?
Dr Oransky: I think there will. I am quite hopeful. It is not just our efforts but the efforts of organisations such as PubPeer, where people can leave comments and analyse papers freely and publicly. That kind of crowd-sourced scrutiny is a longer-term solution than simply looking at particular individual papers. While we are very hopeful that our database of retractions will help people to cut down on waste and inefficiency in research, we think that the database itself is an ongoing proposition. Of course, because we are interested in it, we think that Retraction Watch itself, and the scrutiny we provide, is very important and will continue, but, like many non-profits and public charities, we would be quite happy to put ourselves out of business in such a way that others took up the mantle.
Q284 Martin Whitfield: This question is for Professor Gunsalus. To set the scene, could you briefly outline the role of the Office of Research Integrity and the National Science Foundation in regulating research misconduct?
Professor Gunsalus: Certainly. Ivan referred to two things that I’d like to tie into my comments. One is looking at this with a sense of historical perspective; the other is the importance of transparency and scrutiny. In the United States, the modern history of scientific misconduct started in the late 1970s, and I think you will find that many other countries followed a similar path: a problem occurs; there is a question about whether oversight and regulation would be useful; there is significant pushback, with people saying regulations are terrible and they will do bad things and interfere with scientific progress. The regulation occurs; people normalise it and things stabilise a bit. Then another problem occurs so they look at it again.
In the United States, the product of that is the conclusion that, in terms of scrutiny and transparency, there is no real option but that when a problem occurs the employer is the first person to review what happened. The universities are employers of scientists. The students are on their premises; the research occurred using their equipment and their facilities, and with their HR systems. They are the ones who must review it.
One of the things we know about human nature is that people tend to excuse things that happen in their in-group more than they do with others who are outside. There is a lovely paper by Valdesolo and DeSteno that talks about moral hypocrisy, with a lovely example of how people portray transgressions by people like them, as opposed to outsiders. Since we know that human beings tend to excuse the activities, problems and shortcomings of those like them, we know that oversight and compliance are necessary.
How do we achieve accountability without impeding scientific progress at a reasonable cost-effectiveness ratio? The balance we have come to in the United States, which is far from perfect, involves placing the first level of responsibility on the institutions that receive the funding and where the research occurs. They are accountable to the Federal Government if they accept any federal money. They must have procedures; they must have a named person; they must report annually on their progress; and they must provide education on responsible conduct in research.
Q285 Chair: If they do not report annually, for example, what is the sanction? Can they be fined?
Professor Gunsalus: The sanction is that the entire institution is not eligible to receive federal funding. I am not aware of a single instance in which that has happened, and I do not know how rigorous the compliance and verification process is. I served as the research integrity officer at the University of Illinois for eight years. In that time, we submitted our report late one year, through a bureaucratic issue. I thought it was submitted and it did not go through, and we did get a query about why it had not been submitted.
Every single time an institution conducts an inquiry or investigation in the United States, there is a three-tier system. When an allegation comes in, it is assessed to determine whether it falls under the misconduct in research/integrity policy. If it falls under it and it is a credible allegation, for which some evidence exists or might exist, there is a relatively rapid triage stage that involves an inquiry. It is relatively fast and confidential. There is an assessment of whether it is worth going through a full investigation. The full investigation typically involves an external member of the institution as a check and balance. The institution is obliged to report when it initiates an inquiry and to submit its investigation report.
The two federal agencies, the Office of Research Integrity for the Department of Health and Human Services and the Office of Inspector General for the National Science Foundation, review those reports and have the ability to query them and send them back to the institution and say, “You’ve missed the mark. Do it over again,” which they do with some regularity, before they accept the institutional findings. The two agencies operate under completely different governmental authorities. The National Science Foundation has the authority for subpoena power to gather evidence itself and to initiate its own investigations. The Office of Research Integrity does not. That is a vast overview and I would be willing to go into detail if that would be useful.
Q286 Martin Whitfield: I am very grateful for that. We have been told—I wonder whether or not you agree with this—that the fact that the investigation moves out of the institution later on means that it is less likely that the researchers will learn from their mistakes and will become afraid of them. Does that make sense?
Professor Gunsalus: I am mystified by that. In the United States, the initial process is virtually always conducted by the institution. The home institution retains the employment status and the ability to discipline, and/or terminate if required, an employee; and the ability to impose non-employment sanctions. One of the regular outcomes is that somebody is required to undertake or redo a research integrity course or a course on responsible management or data management. Those are regular outcomes. I am bewildered that having oversight would somehow reduce internal institutional capacity.
Q287 Martin Whitfield: Your experience is that the internal first-level contractual obligations between employer and employee work very well in maintaining discipline with regard to research.
Professor Gunsalus: I would not go that far; the external oversight does not do anything to compromise that relationship or the educational function. There are significant questions about the rigour and integrity of internal university reviews. The quality is extraordinarily uneven across the United States, and it is a gaping hole in our system.
Q288 Martin Whitfield: With regard to cost, which you mentioned before, do you think the system operated is good value for money?
Professor Gunsalus: By and large, we have not always achieved the optimum balance in how to achieve accountability. The next step will have to be considering additional forms of transparency and the kind of after-review that Ivan referred to, with more eyeballs on publications. After 30 years of hope for something different, I have come to the opinion that complete transparency may be the only option for assuring the kind of accountability we need if our institutions are to be trusted by the public, to whom we are ultimately accountable.
Q289 Martin Whitfield: That transparency really lends itself to the self‑regulation of researchers themselves.
Professor Gunsalus: People watching is a very healthy accountability factor. We do better when others are watching.
Martin Whitfield: I appreciate that.
Q290 Chair: You do not feel that the American system is yet working effectively enough to secure the trust you are after in the research institutions.
Professor Gunsalus: We have made progress and we have very clear indicators of the places where more progress is needed, so I do not think we are yet at a level that is acceptable.
Q291 Stephen Metcalfe: I want to explore that point a bit further with both Professor Gunsalus and Dr Oransky. The UK Research Integrity Office told us in its submission: “It would be extremely challenging to establish a body which could regulate all aspects of the research enterprise.” Presumably, that means a unified body to regulate all areas. How does that operate in the US? You have already talked about the NSF and the Office of Research Integrity. Are there other bodies involved in regulating research? Can you ever envisage a system where there would be one body with overarching responsibility?
Professor Gunsalus: The United States has an extremely decentralised system, so for every research institution there is a plethora of regulations. There are animal welfare regulations; there are hazardous substance regulations; there are conflict of interest regulations; there are research misconduct regulations; and there are export regulations. The regulations are myriad.
The home institution is the responsible party for implementing and ensuring compliance with the regulations. For example, in the regulation of research inside a research-intensive university in the US, you are likely to have an animal welfare organisation with an institutional veterinarian; a research integrity officer; an institutional review board that is responsible for the protection of human subjects in research; a hazardous substances organisation, and a grant and contract review. The regulations are many, and in every instance the home institution is responsible for compliance.
Dr Oransky: Professor Gunsalus’s excellent overview did not leave out any details. I completely agree that there is no overarching organisation. Without speaking for the UKRIO, or trying to understand exactly what it meant, in large part I agree that one single regulatory agency or body would be problematic, for exactly the reasons Professor Gunsalus just mentioned. There are so many different elements.
One element that is not a regulatory agency but has become part of the unofficial oversight of research, and is becoming more and more active in this country and around the world, is the role of the courts, both civil and criminal. Alison Abritis and I, from the Centre for Scientific Integrity, our parent organisation, presented an abstract to the World Conference on Research Integrity, looking at how many researchers had been sanctioned criminally or faced criminal sanctions, everything from probation to prison time, for scientific misconduct. We looked at a period of about 40 years. We do not claim to have caught every single case, but we feel confident that we have at least most of them. There were 39 cases; maybe there were a few more here and there, but that is essentially one per year. One was in the UK—a case involving Steven Eaton, which you may be familiar with. This is something that is probably growing, although maybe not in criminal terms. It has happened in Australia a few times.
We are particularly interested in looking at the civil court system. Without being an attorney, a lawyer, I do not want to mischaracterise this, but, very loosely speaking, the UK and US civil and criminal court systems are, roughly speaking, similar, meaning that you understand the delineation. What has become a little more common is the use of the civil courts to defend scientists against misconduct charges. That often comes down to employment law, and defamation law, for example, which is different here from the UK, and something called the False Claims Act. The idea is that if you make a false claim to the Government, including in a grant application, or even in a paper based on that grant funding, you can be held liable.
I will not go into details. I can follow this up or answer questions, but essentially incentives are created by the False Claims Act for people who blow the whistle—for example, people in a lab saying, “There’s something going on here.” There is considerable compensation for that. Major universities here, including Harvard University and its affiliates, and Columbia University, have settled such cases recently, for different reasons, but one of them was exactly for scientific misconduct in making false claims. It is not a regulatory agency, but, looking at the entire picture here, it is important to keep in mind the potential or real role of the courts and the judicial system.
Q292 Stephen Metcalfe: Who brings cases in the majority of instances? Is it universities against their own scientists, or is it the Government saying, “We’ve been misled”?
Dr Oransky: I do not want to oversimplify, but I will a bit. Generally, in False Claims Act cases, a whistleblower, who may work in the lab or in the university, has knowledge of a situation and brings that to an attorney, who then brings the case. The Government then have the choice as to whether or not to join it. It is much more likely to succeed if the Government joins it. They can still succeed if not, although there is a lower likelihood. That is the process.
Q293 Stephen Metcalfe: The whistleblower does not have to take on the cost or responsibility of doing it; someone else will do that for them.
Dr Oransky: Correct. The whistleblower may do that, and there are cases where they have, but in the ones where the plaintiff prevails, typically the Government have joined and have done a lot of investigation that we do not know about at this point.
Professor Gunsalus: I would like to broaden the picture. The whistleblowers bring concerns because they believe their concerns are not being heard. Typically, they try internally and get rebuffed in some fashion at an early stage of the process that is particularly subject to governmental oversight, and somebody says, “This is not an allegation worthy of review.” We also see lawsuits by individuals who have been found by their university to have committed misconduct, and sue to have it overturned. People use the courts as a last resort when they believe that the institutional processes are not fair. It is a much more expensive and cumbersome system, and it strikes me that if the initial system was perceived to be fairer, with a more level playing ground and fewer conflicts of interest, and if it was more properly administered, we would have less dependence on the legal system.
Q294 Stephen Metcalfe: Looking outside the US and the UK, are there examples of systems for regulating research integrity that you envy and you think work better than the system you have?
Dr Oransky: I don’t think that regulating is the right word, but I take your meaning. I would be interested to hear what Professor Gunsalus says about this. For me, the best system would be to pick the best from each and avoid the worst in each. I will mention briefly two different systems. In Japan, there is a legal obligation that, when there is a finding of misconduct, the university must issue at least an executive summary. They often issue a fairly substantial investigation report. We think that is good for transparency. Personally, I would like to see them release details when there is not such a finding. Of course, I understand that one is innocent until proven guilty, and all those considerations.
It is worth mentioning Denmark and Sweden, but I will focus on Denmark. Until July, it had something called the Danish committees on scientific dishonesty. That is the English version; in Danish it is something else. The acronym is UVVU. Again, I am going to oversimplify. This was an agency outside the universities that had some authority. I believe it was most recently chaired by a retired judge—someone who knew the systems and the legal framework and could adjudicate. My take on it is that they had some initial successes in adjudicating some issues. They made some misconduct findings, and what I would loosely call sloppiness findings, but in at least several of the high-profile cases that we covered and have been following they were rebuffed by the courts; in other words, the fact that they did not have the supreme authority that a Government agency might have, because they are not exactly the Government, meant that their effectiveness was limited. That was part of the thinking when they changed the law, and the name of the organisation as well, in July. New laws came on to the books.
I do not speak for scientists, but I know that a lot of them are concerned that taking this out of the universities and putting it into the hands of, say, a Government agency or even an external agency would be problematic, because they feel that those agencies might have other concerns and priorities that they do not share. Somehow, though, we need to take it out. Denmark is one place that is definitely looking at that, and Sweden too. I will not take more time to discuss it, but that is another case of a non-Government, but outside, agency looking at it.
Q295 Chair: Do you think the threat of loss of grant funding and access to public grants for the institution is in some way an effective deterrent to bad behaviour?
Professor Gunsalus: At institutional level, it is an extremely powerful threat. For example, in the United States there were a number of circumstances over 10 to 15 years where the human subject oversight agency shut down the ability of a university to conduct research with human subjects. That brings extraordinary focus and attention on how the university improves its internal oversight. Because of the size of the stick, it is rarely deployed and there is often a massive pushback against it. On the other hand, the threat is quite significant.
People are busy. Everybody worries about multiple things all the time, so, given human nature, what rises to the top of the heap are the most urgent and pressing problems—this is on fire, so we have to go deal with it—and all the other things don’t get attention. The challenge is to build systems that have sufficient checks and balances, accountability and transparency, so that the normal state is a positive state that reinforces all of people’s better angels, as opposed to providing incentives and rewards for bad conduct. The challenge in all these systems is how, in the constant press of daily business, you keep the focus on the aspirational standards we all need to achieve to retain trust, and do rigorous, replicable research that others can trust and build upon.
Q296 Bill Grant: Dr Oransky, thank you for the good work you have done on Retraction Watch since 2010. I note the mention of civil courts and whistleblowers, but in your written evidence you suggested that a national oversight body could play a role in the United Kingdom in investigating and penalising scientific misconduct. Do you see that approach as a universal solution—for example, something every country should consider—or does it relate particularly to the United Kingdom, and would it be opposed to self‑regulation, or self-policing, as I would describe it?
Dr Oransky: I was submitting to this particular inquiry, but we talked about the fact that it was a good idea for every country to consider. I do not think there is a magical or perfect system we could point to and say, “Just do that.” There are elements of systems in each country and lessons to learn from countries that have not got it quite right. It is a universal need.
Professor Gunsalus: There was a fascinating piece by Drummond Rennie, who was originally British and is now American, published in the British Medical Journal. It is called “An American perspective on research integrity, dealing with misconduct in the United Kingdom.” I pulled that this morning and read it; it is volume 316, June 1998. It lists the important procedural criteria and some of the rationale and the history. I commend it to your attention. It is a really lovely piece, and it is as applicable today as it was when it was written.
Q297 Bill Grant: Thank you for that additional information. The Association of Research Managers and Administrators in the United Kingdom told us: “There is little evidence to suggest that regulation has had a significant impact in other countries.” Is it possible to point to hard evidence that a regulatory system—for example, in the United States, Denmark or elsewhere—has improved research conduct, or reduced, or evidenced a reduction, in misconduct in research?
Professor Gunsalus: May I ask a question in return?
Bill Grant: Certainly.
Professor Gunsalus: Is there any evidence that having a courts system and a system for dealing with arrested miscreants has reduced the rate of crime in the United Kingdom?
Bill Grant: I think that will be one each.
Professor Gunsalus: The issue is what we know about checks and balances leading to scientific credibility and improvement, sincerity and the individual being honest. What we have discovered over the last 40 years, and longer, is that some small and probably irreducible number of human beings are going to commit misconduct. The question is how easy we make it. How many checks and balances are there and what deterrents do we have? What consequences are there if you are caught misbehaving? Do we align the system so that people are more likely to take short cuts and cheat because they get rewarded for it, or do we align the system in a way that there are sufficient checks and balances, so that somebody who is well meaning says, “Oh, the risk is too high; I’ll get caught out. I don’t want to do that”? Then we have the people who would never get involved in bad practice.
It is the middle people we have to design the system for, the people who say, “What are the incentives? How likely am I to get caught? What are the rewards? How pressured am I?” We have to design a system that says, “Go with your better angels and stay away from those bad practices.” There is an irreducible number of people who will always try to cheat. It is the place in the middle that we are trying to reach and to urge better conduct, through a combination of education and oversight, and deterrence, by saying, “Bad things are going to happen if you get caught.”
Q298 Chair: None the less, we can at least seek to compare countries such as the States, which have a system of regulation in place, with other countries that do not. Is there evidence you can point to that suggests that by having your system of regulation you have managed to have an effective deterrent that has reduced the amount of misconduct in your jurisdiction compared with others? Are you able to say that or not?
Professor Gunsalus: I do not think we can say that. The question is about reducing the incidence of misconduct, surfacing it when it occurs and responding to it. Human nature is human nature, so the question is: what happens when somebody mis-steps, rather than how are we going to prevent everybody from mis-stepping? It is certainly to do with education about how we apply statistical practice appropriately. How does a responsible scientist or researcher deal with data? How does a responsible scientist or researcher deal with publication ethics? A certain amount of baseline professional knowledge is needed. Then there is what happens when someone mis-steps. It is a continuum.
Dr Oransky: I agree with Dr Gunsalus. We are dealing with very different use cases and situations on a spectrum. One of the things that Adam Marcus and I urge everyone to look at is encouraging good behaviour using the right incentives. This may not prevent people who are determined to commit fraud or those who will always commit misconduct, but, as Dr Gunsalus put it, it will get at the central group.
One of the problems we have is that most incentives are, I would argue, aligned in the wrong way. They are aligned in such a way that the only reward is publishing in journals that have what is known as a high impact factor, which we are all aware of. In situations like that, it is completely predictable that a percentage of people will cut corners, and perhaps do even worse than that, so we need to reward good behaviour. At Retraction Watch, we have the annual Doing the Right Thing award. It is almost a fun thing, but perhaps we can create incentives that are not completely reliant on one metric—incentives to share data and correct the record in publishing and so on. We can do some things along the lines of education and preventing, but we will still need sanctions and some sort of process for what happens when it goes wrong.
Professor Gunsalus: There are two kinds of incentives for good conduct, at the individual level and at the institutional level. Retraction Watch, in its database, is both recording and creating accountability for whether publications and the people citing them are bringing attention to the retraction. That is at the publication, the individual, level.
At the institutional level, we have an initiative based on the work of Brian Martinson and Carol Thrush, which is the only validated instrument that measures institutional climates for research integrity. It is called SOURCE—a survey of organisational research climate. We are creating a national database that measures institutional integrity in terms of the research climate, because we know that the decisions people around you make profoundly affect and influence their choices about the integrity of research. If you administer the SOURCE 32-question survey about your institution, you can find the bright spots in your institution, with exemplary, first-class data, to place education and integrity norms and integrity socialisation. You can also find places that are less optimal.
Two things happen regularly. Institutions say, “What if we administer the SOURCE and find that our research climate is terrible and not full of integrity?” But that is better than if they don’t find it out. It gives them a metric that is validated, with 40,000 respondents, for finding it out. The other thing is that a department will look at the results and say, “Our results show very low for the research integrity climate in our unit, but every department is different according to the discipline.” One of the arguments against any of these things is that the disciplines are so different that they cannot possibly be regulated. By creating an international database, we can say that at other universities there are very high scores—
Chair: That is really helpful. Thank you.
Q299 Bill Grant: To date, UK Governments have not been minded to set up an overarching regulatory body in the United Kingdom. If they were minded to set up a regulatory body, what problems do you think that body would solve, if indeed any?
Professor Gunsalus: I think what you would solve with a regulatory body would be holding institutional leadership accountable for the integrity climate in their institutions and for ensuring that their processes are open, free of conflict and credible. We find that internal investigations tend to have pretty significant shortcomings if there is an oversight, and I think they should apply even-handedness, with recourse for individuals when they have not been treated fairly.
I do not think you want them to be first responders; on the whole, you want them to be empowered, and to have a process with consistent metrics. One of the things said in business is that, if you don’t measure it, it doesn’t matter. We need to measure. How are we doing with the institutional climate in a positive sense? What are we doing to respond to problems? How credible is that? People need to say, “Have you asked the right questions? Have you involved the right people? Did you actually examine the data? Is this a credible outcome based on the data that have been assembled, in the case of a problem?” I also believe we need to measure the internal integrity climate. It is the job of leadership to maintain that.
Q300 Bill Grant: I get the sense that to a degree you are an advocate of some sort of regulatory level across the board. Can I take you to the Australian model? From memory, it is almost like a court of appeal or an independent body that deals with issues that may emerge from research establishments, universities or privately funded ones? Do you see merit in the Australian model that would take up the slack or take up your concerns?
Professor Gunsalus: Some of it is cultural; it relates to national culture and what would be accepted and effective. One of the things cited in the statement of Universities UK surprised me; they said that 85% of universities have accessible policy documents related to these topics, which says to me that 15% of your universities still do not have accessible policy documents. That struck me as astonishing.
Q301 Martin Whitfield: Again, this is probably to Professor Gunsalus. You talked about criminal prosecutions. We heard earlier that around the world there have been approximately 39 cases, or one or two a year. How does the research community in the US feel about the risk of criminal prosecution and conviction with regard to their research?
Professor Gunsalus: I think it is so remote that it does not really exist in most people’s brain. Every now and then, an occasional case arises. One of the serious problems is that people who attempt to seek redress outside an institution, when the institution does not provide it, frequently find that the legal system cannot provide any recourse. I am a lawyer so I am a little familiar with this. For example, in the United States if there is sabotage of an experiment people frequently say, “Oh, you should go to the local prosecutor because that’s vandalism. You should have legal recourse if somebody vandalises your research.” It is not realistic that anyone would prosecute that in the United States because you do not have the expertise and the resources to do it. A handful—a very small number—of researchers have gone to prison for, typically, misuse of very large amounts of federal funds, typically fraud, and I think most people rationalise it to themselves as, “If I wind up getting caught, they’re not going to touch me anyway.”
Q302 Martin Whitfield: Do you think it is detrimental that there are so few successful prosecutions, or is that not the way we should look with regard to research integrity?
Professor Gunsalus: It should be the extreme last resort. If internal systems fail, it strikes me that an institution administered and populated by highly educated people, devoted to creating and expanding knowledge, should be able to devise reputable, credible systems for ensuring that their mission is met, having tested out systems for the natural failings of human nature. We ought to be able to do that and we have enough information to do it.
Q303 Martin Whitfield: Is there a blacklist of researchers that institutions will not touch because of their track record?
Professor Gunsalus: Unfortunately, in the United States, given the way the laws are working, a lot of institutions are regularly passing their trash from one to the next by providing very limited references. Other institutions do negligent hiring, in that they do not necessarily check references carefully. In preparation for this, I went back and looked at some things, which is why I found the 1998 paper by Rennie. In 1982, two New York Times reporters wrote a book called “Betrayers of the Truth: Fraud and Deceit in the Halls of Science.” I looked at it over the weekend. It is fairly up to date and accurate all these years later. There are two chapters about recidivism and how people go from one institution to the next. I am familiar personally with five cases where individuals who were found to have committed research misconduct in the United States moved to the United Kingdom and are active researchers there, and vice versa—people who got into trouble in the UK and moved to the US and started anew. Recidivism is a fairly serious problem. We do a very poor job of reference checking. Modest reference checking typically surfaces these problems. Any institution that covers up a problem in order to get rid of somebody ought to be shamed everywhere and reviled.
Dr Oransky: I second what Dr Gunsalus has just said. It is even more global than that. She was looking at two particular countries, but we have seen a number of cases where people in the US or the UK have gone to other parts of the world. Even within the United States, people go from one state to another, as she mentioned.
The blacklist is not at the level of federal or Government sanction, or even funding sanction. Publishers, particularly large publishers of journals, have told us that, although they do not make the lists public, they have created their own blacklists, often because of plagiarism as that is something they can check easily. They are out there, and there are publishing bans. I wanted to share that for all the reasons that Professor Gunsalus mentioned in terms of the law and some of the risks. People are thinking about that. Whether they are making the lists public or imposing any sanctions is another question.
Professor Gunsalus: The federal funding agencies in the United States, the National Science Foundation and the HHS maintain internal databases that they check before awarding funding. I am not familiar with those systems, but I understand they exist.
Q304 Martin Whitfield: Does it concern either of you that there may be secret lists of researchers who are not getting funding or published?
Professor Gunsalus: This is a quote from the Rennie paper: “Legitimate institutions, whether or not they were funded by the public, also depend on public confidence, and public interest requires that the process and resolution of cases be made public.” I think we are acknowledging that we all have to come to terms with that.
Q305 Martin Whitfield: Do you think we have reached a point where there needs to be almost a global licence to research, which could be logged and checked?
Professor Gunsalus: I am not there. I am more towards sunshine on the existing systems.
Dr Oransky: I would have concerns about that. There should be a global oversight system of some kind; it would be great if you could work that out. Licensure is always tricky. I say that as someone who trained as a physician in the States but does not have a licence. I never practised medicine, but I practise journalism. Those two use cases are so different, and I would put science more in the journalism use case. Physicians are doing things that could kill people. Scientists are indirectly doing things that could and do hurt and maim people, and some research does that. However, the practice of research in science is much more akin to something that needs to be a bit unfettered. Institutions should have assurance agreements, as there are in the States and elsewhere, with the funding agencies in order to get funding, but submitting to some sort of licensure for practising research would create more problems than we solve.
Chair: We are running out of time.
Q306 Vicky Ford: I am Vicky Ford and I have a short period of time so I ask you to be really brief. They are very broad questions. In this inquiry, we are looking at which integrity matters the research community should be dealing with and what Government should be dealing with. Are there any other countries that you think we should look at where they are doing this particularly well, or is there anything we should learn from other parts of the world?
Professor Gunsalus: I would endorse the earlier answers about Denmark and Japan.
Dr Oransky: I mentioned Sweden only in passing, but they have had some recent, high-profile problematic cases that were not dealt with well, which is often what gives rise to reform. What they are doing there is very thoughtful, and I would recommend that the Committee takes a look at some of their processes.
Q307 Vicky Ford: Thank you; we will do that. Given that science is increasingly collaborative, does that bring benefits, because it makes it easier to spot a rogue, if I may say that, or does it help to hide ideas? What are the pros and cons of the collaborative nature of science?
Dr Oransky: It is very much both. One of the problems is that many clinical trials are carried out in multiple centres around the world, and sometimes the standards are different and it is hard even to follow what is happening. It is difficult to have an audit trail. That is a cautionary tale. On the other hand, collaboration is not necessarily a matter of groups working together directly, but of science writ large and the ability to look at all papers; for example, I mentioned PubPeer earlier. That is very powerful, not opposing forces but just different forces.
Professor Gunsalus: I concur.
Q308 Vicky Ford: We are about to hear evidence specifically about clinical trials, encouraging us to say that all trials should be made public, because, if you are reporting only those that have positive impact, and not trials that have a less positive one, you are missing half, or more than half, the information. Is that what happens in your country? Do you have that AllTrials approach?
Dr Oransky: I know that AllTrials has branched out as an activity in various countries, and I think that is terrific. The problem with the positive publication bias that you have just described is pervasive, not only in clinical trials but in science writ large, particularly in the basic sciences and life sciences.
Q309 Vicky Ford: Are you doing anything on that in the States—that AllTrials-type approach?
Dr Oransky: Others are. Professor Gunsalus may know more about this than I do, but we are not directly involved. We have enough fish to fry, but we have seen lots of people take this by the horns, both academics and in organised medicine. We have ClinicalTrials.gov and there are international versions of that. It is a repository or registry of all trials, so at least we have to register and can say, “This is what we are going to do.” I think you will hear more about that in the next session. At least there is an audit trail. The FDA is now being put on notice that they could fine researchers or institutions for not including all the data, and not actually publishing or posting all the findings, only the positive ones. They have not done that yet so there is not actually any movement, but they could do that.
Professor Gunsalus: ClinicalTrials.gov is an interesting example of internationalisation. The existence of ClinicalTrials.gov in the United States has driven that in other countries. It is an interesting model. That internationalisation will continue and we need to build it into our system.
Q310 Chair: Finally, is there any resistance from universities and other research institutions as to the framework of regulation in the States, or do they accept it as necessary and legitimate? You are laughing. What does that imply?
Professor Gunsalus: The universities and research establishments in the United States have resisted all regulation since the beginning of time. In 1992, Paul Friedman wrote, “The argument that science must be regulating itself pretty well because it is making progress is far from compelling, perhaps progress would be twice or four times as fast with greater ‘scrupulousity’.” We are struggling with this. Universities say, “Nobody wants more regulation; it’s just more overheads. Regulations are costly; they are cumbersome; they require a bureaucracy to implement.”
We are seeking a balance. Nobody wants to be regulated, yet we have not reached a place where we can trust institutions to do it themselves, so some oversight, accountability and sunshine is necessary. The recent National Academies panel report “Fostering integrity research” advises that there should another body, the research integrity advisory board, to help institutions do better, because their shortcomings are so clear across the United States. It is starting to look again at a voluntary system, but voluntary systems, unless they have some teeth, have not so far proven to be effective.
Chair: Professor Gunsalus and Dr Oransky, thank you both very much indeed. We have enormously appreciated your time in answering questions.
Examination of witnesses
Witnesses: Dr Goldacre, Dr Kolstoe and Síle Lane.
Q311 Chair: Welcome, all three of you. Thank you very much indeed for coming along. Could you first introduce yourselves briefly, saying who you are and where you are from?
Dr Goldacre: My name is Ben Goldacre. I am a doctor. I write books about problems in science, including research integrity issues, and I run something called the DataLab in the department of primary care at the University of Oxford.
Dr Kolstoe: My name is Simon Kolstoe. I was originally a biochemist working in pharmaceutical drug design and became very interested in research ethics. I am now an academic at the University of Portsmouth in research ethics. I chair an NHS ethics committee and a Ministry of Defence research ethics committee.
Síle Lane: I am Síle Lane, and I work at Sense about Science, which is an independent campaigning charity that responds to the misuse of evidence in public life. At Sense about Science, part of my job is running the global AllTrials campaign for clinical trial transparency. We call for all clinical trials to be registered and the results reported.
Q312 Chair: Excellent. We have quite a lot to get through in the next hour, so please try to keep your answers succinct. There is no obligation for all of you to answer every question. If you feel you have something to add, please come in, but don’t feel you have to.
Your submissions have highlighted the issue of clinical trial transparency. Can you explain how that relates to research integrity?
Síle Lane: The research that has been done into clinical trial transparency, the solutions now being mooted and the work being done are a really good model for how researchers interested in other areas of research integrity and publication bias could learn from the things we have been through and have done. Clinical trial transparency is the most researched area when it comes to research integrity and publication bias.
Publication bias in clinical trials was first identified 50 years ago; it has been written about and discussed in more detail for 30 years. Over the past two decades there have been hundreds, maybe thousands, of pieces of research just into publication bias in clinical trial results, looking at different subsets of trials, such as trials of different treatments done in different regions by different sponsors, to try to identify which trials have been run, which have published results, or not, what kinds of results they are and what patterns and problems there are.
To fix the problem, a necessary first step is putting some numbers on it, so that we can see how it is happening and where. The unique thing about clinical trials, which has allowed all these hundreds of pieces of research to be done and allowed people like us, who are concerned about clinical trial governance and so on, to be involved in part of the conversation about fixing this, is that clinical trial registers exist. They were mentioned just at the end of the previous session. Those registers are publicly accessible databases of information about clinical trials being run, whether set up by Governments, funders or other institutes. Their primary purpose was to increase participation in clinical trials, hence the WHO’s mission for every country to have a national clinical trial register with information in that country’s language so that local patients and doctors can review what trials are going on and perhaps join those they are eligible for.
A second function, which we talk about a lot more, is that clinical trial registers allow scrutiny; they encourage accountability; they allow researchers interested in publication bias to see at least some of the trials that have happened and follow them up and see whether they have published results. They allow people who make decisions about treatments, be they patients, doctors or researchers, payers or regulators, to scrutinise what trials have been done and make sure they have full information.
Q313 Chair: Brilliant. Are there contributions from either of the other witnesses?
Dr Kolstoe: Ben made an interesting point in his written submission when he said fraud is not the most important issue. I am involved in something called the REWARD Alliance, which stands for Reducing Waste and Rewarding Diligence. We have seen statistics showing that up to 85% of research funding is wasted because it asks the wrong questions, is badly designed, is not published or is poorly reported. One of the big issues around integrity is the wider culture of science. It is all well and good getting funding and doing the experiments, but are those results getting out in a way that is usable? That extends beyond clinical trials. These guys will focus very much on clinical trials. I am really interested in the whole gamut of science. It is a waste of funding if results are not getting out in a usable way, and I think that is a key integrity issue.
Dr Goldacre: Clinical trials are a great thing to focus on because they are very important. There is good evidence that they are broken, and there are things this Committee could do to fix that very straightforwardly. I am a doctor. They are important because doctors cannot make informed choices about which treatment works best without all the results of all the clinical trials that have been conducted and completed. We know from the best currently available evidence that about half of all the trials conducted and completed do not go on to report their results, and we cannot make decisions when doctors, researchers and patients are deprived of that information.
Q314 Chair: Are there potentially serious consequences from that?
Dr Goldacre: Yes, absolutely. It is reasonably uncommon that there are treatments on the market in widespread use that are worse than useless, but it is very common for us to be bounced into using the second, third or fourth best treatment in class because of withheld information about which is the best.
Q315 Chair: It is wasting a lot of public money and not achieving very much.
Dr Goldacre: Yes, and wasting public money on conducting trials and then not reporting the results of publicly funded clinical trials. This is all very simple to fix. When it comes to social science research or molecular biology, there are thousands, probably millions, of different study designs. People are doing different experiments on the hoof every day in their own laboratories, behind closed doors, and it would be very hard to track, but with clinical trials, as Síle said, there are clinical trial registries. You have to register your trial in order to conduct it; you cannot do a trial in secret very easily in this country because, obviously, you have to recruit patients through clinics and hospitals. We know about all the trials that are done, and it would be very straightforward to set up a clinical trials observatory nationally and say that every trial that is given approval by an ethics committee has to report its results. We could say that every clinical trial that is conducted in a university has to report its results.
Q316 Vicky Ford: Slow down. What was your first example? You said every trial.
Dr Goldacre: Every trial that gets permission from an ethics committee should report its results, and we should require ethics committees to get an end-of-project report where they say whether or not the trial has reported its results, and disclose that publicly. Every time a trial is conducted in a university in the UK, we should make it a requirement that it reports the results, and we should require UK universities annually, or perhaps as part of their REF return, to say, “Here is a list of all the trials we’ve conducted, here are their completion dates and here are the results.”
Q317 Chair: What would be the sanction for not doing so?
Dr Goldacre: It depends on who is doing the audit. In the case of ethics committees, it would be very reasonable for an ethics committee to say that, if you are applying for permission to conduct a clinical trial in patients, you should be required to sign a piece of paper saying, “Here are all the trials I have ever conducted, and here are all the results.” Ethics committees should not allow researchers to have access to patients unless they can show that they have published the results of all the trials they have previously conducted. Withholding the results of clinical trials is unethical, and it is not appropriate for ethics committees to permit people to go on to do more trials if they have already completed trials and not reported the results. Similarly, funders should be required to audit line by line.
Q318 Chair: We are hearing from the States that the institution runs the risk—the nuclear option—of not getting any public research funding if there are breaches of ethics rules.
Dr Goldacre: As things stand, there are no consequences for breaching the obligation to report clinical trials in the UK. In many cases, people will never even know that you did that, because nobody is doing audits, other than our team. We have a whole series of audits; one or two are published, and one or two are coming down the track, which we can discuss later.
Having run a number of audits looking at lists of trials that have been conducted and completed and seeing which ones have results and which do not, it is striking that there is nowhere to go with the results. You can publish a paper or you can build a website where you share lists of trials. A junior doctor, as I was for many years, is required to do audits all the time for quality improvement. You do audits of infection rates in your surgical unit; you do audits of waiting times for talking treatments for depression and so on. When you finish your audit, there is a hungry audience waiting to hear. People are not delighted to hear that they are not doing perfectly, but theatre staff want the audit on the infection rate in the operating theatre and the people running outpatient CBT want the waiting time audit.
When you finish an audit showing which institutions in the UK are best and worst at reporting their clinical trial results at all, and which are best at reporting their clinical trial results on time, people sometimes respond as if you are doing something that somehow is transgressive or confrontational, which shows how far we have to go. It would be a very straightforward thing to fix. This Committee could write to the Health Research Authority and say, “We want you to audit every trial that you approve; we want you to publish line by line; we want you to identify the individual trials and trialists who have not published their results.” You could write to the MRC and NIHR and say, “We want you to identify every trial you have funded for the past decade and identify whether or not the results are published.” You could write to the people who run the REF and ask them to make that part of the process of judging the quality of institutions, which we do every five to seven years.
There are already many bits of paperwork associated with clinical trials. People who run clinical trials are rule followers. Running a clinical trial is a horrible, entangled, administrative process, and a lot of the regulations are overdone and inappropriate, but the one thing we do not ask people to do is simply to say, “Here are the results of my clinical trial.” It would be a very straightforward thing to ask people to do. I can guarantee that, if we asked trialists to do it, they would, because clinical trialists are rule followers. That is what they do; they follow rules to get trials done.
Q319 Chair: Our predecessor Committee last looked at transparency in 2013. What, if anything, has changed since then?
Síle Lane: Lots has changed since then. This has now become a discussion at the highest levels, and international and intergovernmental organisations, including the WHO and UN, have taken it up. Last year, the UN urged all Governments in the world to bring in laws to mandate clinical trial registration and results reporting in their own country. The WHO made a very strong statement in 2015 that public disclosure of results of clinical trials is an ethical imperative for researchers. Just a few months ago, it published a very strong policy, and asked global non‑commercial clinical trial funders to sign up to it. The policy says that a funder will ensure that all clinical trials it funds are registered before they begin, that results from them are published, and, importantly, that funders who join in that statement will find a way to monitor and track it.
Q320 Chair: Who has signed up to that?
Síle Lane: Right now, 21 groups, including Médicins Sans Frontières, the Bill and Melinda Gates Foundation, the Wellcome Trust and Government funders—the NIHR, the MRC and global MRCs.
Q321 Chair: Those British funders have signed up, yet they are not requiring this.
Síle Lane: They signed up three months ago and said, “We will produce this very strong policy in a year’s time.” However, some of them already had a public policy that they expect researchers to register and report results.
Q322 Chair: The commitment is ahead of the reality.
Síle Lane: Yes. A few years ago when MRC and Wellcome were here, they said they would do an audit, didn’t they?
Dr Goldacre: The last time NIHR and MRC were here, in 2013, to discuss this, they said they would do an audit of all the projects they had funded in the preceding couple of years. They reported the summary results of those, but they did not identify the individual trials that had not reported results and, to the best of my knowledge, they have not repeated those audits.
Q323 Chair: We could ask them to publish every trial that has not reported the results.
Dr Goldacre: Yes, and it is extremely important that that instruction comes from somewhere outside the organisation. It is not that they need to be told what to do because they do not want to do the right thing; it is because people would feel anxious about identifying individual researchers, quite incorrectly. People are concerned that that seems impolite or confrontational, but it is just a simple matter of good audit. We need to know which trials have published results and which have not; we need to know which researchers are the best and the worst and which institutions are the best and the worst, because that is information we can act on. Assuming good faith, I hope that the institutions that have fallen behind would want to learn from those doing well at reporting their clinical trials.
Dr Kolstoe: That solves only 40% of the problem. Much like the evidence you heard just now, you are talking of public funding, but 60% of research funding comes from industry. We need to know about the industry’s trials as well.
Q324 Chair: We should be asking them or requiring them to meet the same standard.
Dr Kolstoe: We should be asking more broadly, and that is where ethics committees are absolutely key. In this country, ethics committees are a point in time when they see all the documentation and know all the trials that are going on. They have access to everything across the board; it is far wider than just the funders.
Q325 Chair: What about the willingness of journals to publish negative results? Has that changed at all?
Dr Goldacre: There is absolutely no barrier to publishing negative results.
Q326 Chair: There may be no barrier, but do they do it?
Dr Goldacre: The barrier really lies in the hearts and minds of researchers. Researchers want to get their results into the most glamorous high-impact journal they can. They start with the New England Journal of Medicine, Annals of Internal Medicine, JAMA and The Lancet, which are the big journals with a very high impact factor, because that is good for their careers. Then they move down the cascade until they get to a journal that will publish their results.
Q327 Chair: But the bias of more positive results than negative ones being published in journals is down to researchers not submitting negative results for publication, rather than anything on the journal’s part. Is that right?
Dr Goldacre: The best evidence currently available shows that it is not journals that are the bottleneck. In any case, journals are increasingly irrelevant in this. Any trial conducted in the US is covered by the FDA Amendments Act 2007. They are required to post their results in structured data tables, in tabular format with brief summaries, online on a website called ClinicalTrials.gov within 12 months of completion. The International Committee of Medical Journal Editors, which represents the big journals of the world, has been clear that it does not regard that as prior publication. Journal publication should not be the bottleneck, and it is not really an issue. I do not need a trial to be published in an academic journal; I need to report its results to ClinicalTrials.gov or the European Union clinical trials register, and that is a very important issue.
Since 2014, there has been a guideline requiring all trials of medicinal products conducted in a European country to report their results, within 12 months of completion, on the European clinical trials register. We have a paper currently under review, in what I hope are the final stages, with the British Medical Journal, in which we look at the rate of compliance with that. We are publishing the data interactively so you can see which universities are doing well or badly in the UK and across Europe. I can tell you that the rate of compliance with the legislation by drug companies is extremely high. That shows that when you give the pharmaceutical industry, in particular, very clear rules, rather than talking about culture and culture change, it complies. Compliance rates in academia are extremely poor but very variable. Some universities are up at 75% reporting; many UK universities are down at 10% or 0% reporting.
Q328 Chair: You think they should be called out on that.
Dr Goldacre: Yes, absolutely. There is a very worrying problem on the horizon—not to talk generally about Brexit—in that there is a new clinical trials regulation.
Chair: We will come to that.
Dr Goldacre: We should not be exempted from that; we need to make sure we keep within it.
Q329 Vicky Ford: I think you have already answered some of my questions. The first one is whether there has been enough progress since 2013. Clearly not. I want to drill down a bit more on some of your suggestions about finding areas where we could make some progress. You said that every trial that has been given permission should report, and every trial in a university should report. Do they need to report in as much detail as a successful trial? Presumably a trial could be stopped halfway through. Could it have some sort of short-form report?
Dr Goldacre: A trial that is terminated is still required to report results. Apart from anything else, they should report on why they terminated; there may be an ethical or a practical reason that others could learn from. When there are trials with unflattering results, or trials that show no difference between the new treatment and the old one, or the two best currently available treatments, that is not a negative or uninteresting finding; it is an actively interesting and important finding that we need in order to make procurement choices, as the NHS, and treatment choices as doctors and patients.
All trials should report their results. The basic filling in boxes on a form approach to reporting trials, which is what you are obliged to do for ClinicalTrials.gov, and for compliance with the European clinical trials guidelines and forthcoming regulation, is absolutely fine by me. There is evidence to show that reporting your trial results in structured tabular format on ClinicalTrials.gov is more complete than, and discrepant with, reports in academic journals. Academic journals are often rather bad places to report the results of science, because they have an interest in overstating things and tend not to require people to follow reporting guidelines.
Q330 Vicky Ford: You said that under European legislation the obligation to report these trials exists and a lot of private companies are complying with it, but it may be that publicly funded university-type organisations are not.
Dr Goldacre: Yes. I would be happy to share a copy of our paper in advance of publication, if that would be useful. In it, we identify individual universities that are doing well and badly. It is a guideline with no penalty, but it is a guideline that the pharmaceutical industry has complied with extremely well over the past 12 months since it came into force, and it will become a regulation, with penalties for transgressions, after 2019.
Q331 Chair: Do either of the other two witnesses have any contribution on this?
Síle Lane: It is fair to say that companies may be a decade ahead, or perhaps less, in thinking about this. Because the law has existed in the US and in Europe for a little while, their regulatory and legal departments have had to be ready to respond. Shareholders and investment groups who have holdings in pharmaceutical companies and have joined the AllTrials campaign—there are €3.5 trillion worth of assets under management from pension funds and so on—are starting to tell these companies, “We need you to be more transparent; we want you to report all the results of all your trials so that we know what the company is truly worth.”
Q332 Vicky Ford: The European legislation exists and will come, with penalties, in 2019, provided we get it in the UK.
Dr Goldacre: Legislation exists solely for trials of medicinal products, so that means drugs and vaccines. There is no legislation at all requiring you to report the results of trials of public health interventions, stop smoking interventions, surgical treatments, infection control regimes in hospital, and comparing one with another; and no requirement to report results of trials of talking treatments for schizophrenia or depression. We have a huge regulatory blind spot, but I do not know that we need penalties that fine people or put them in prison. It would be enough to have very clear data on who has and has not reported the results of their previous trials, and to act on that. Ethics committees should not allow researchers to continue to have access to patients to conduct clinical trials if they have proven themselves to be unethical by not reporting results in the past.
Q333 Vicky Ford: You said that, when people go for their next application, they should have to say, “This is everything I have ever done research on.” Is that practical? Should there be some sort of grandfathering period?
Dr Kolstoe: Clinical trials make up 16% of participant research in the NHS, so the other 84% of research involving participants is currently a blind spot, as Ben said. I argued in the BMJ about a year ago that research ethics committees were in a very good position to demand the information from researchers. One of the criticisms that came back is that researchers move between institutions. The chief investigator on one project may step down and someone else may become chief investigator. Thus, it might be a role for sponsors when they submit a new ethics application, for instance. The sponsors are essentially the people who have legal responsibility for the trial, either a company or a university. They should be the ones to declare all the studies they have sponsored, and whether or not they have published, rather than the individual investigators having that responsibility. There is a certain amount of discussion as to who is in the most appropriate place to make those disclosures.
Q334 Vicky Ford: When the Minister, Jo Johnson, comes in, what specific action should we be asking him about? What questions should we be asking him?
Dr Goldacre: First, why don’t we already have a system in place to identify which trials conducted in the UK have reported and not reported results? Secondly, does he support one? It requires guidance from elected representatives to tell the HRA, the MRC and universities that they should disclose information. Lastly, on a practical matter, what funding will they make available for people to do this kind of audit?
As far as I can tell, almost every single person who has come through this room to talk to you on the issue has addressed research integrity out of the corner of their eye, as a hobby. Ivan Oransky is a journalist; he set up Retraction Watch as a hobby. I am a doctor. I do research, but I set up AllTrials with Síle in my spare time. Simon has drifted into doing research on research ethics, but it is not his principal role. There is no funding for this kind of work, and that is why the problems continue.
Clinical trials are the place to fix it. When I look at things like the concordat on research integrity, I see a lot of motherhood and apple pie. It all sounds really nice, but when I read it, perhaps as a rather simple-minded clinician, I think, “How would I audit compliance with this?” There are no concrete commitments in the research concordat where I could say, “This institution has complied and that institution has not.” That is where things fall down. With clinical trials, it is the canary in the cage; it is the simplest testbed for fixing research integrity problems, and if we cannot fix it for clinical trials we cannot fix it for anything.
Q335 Chair: We must not stop at clinical trials.
Dr Goldacre: I agree.
Q336 Chair: Are you suggesting that the concordat needs to be tightened up, so that there are clearly auditable obligations on institutions?
Dr Goldacre: My principal concern about all the policies and frameworks for research integrity that I ever see is that they are vague. The best proof I have that they are ineffective is that with clinical trials it is extremely simple; it is very black and white. We know the trials that have been conducted; we need to find out whether they have reported.
Q337 Chair: You are saying that we should use that as a testbed.
Dr Goldacre: Yes.
Q338 Chair: Get that right and then learn from other areas of research.
Dr Goldacre: Simon is absolutely right that most other research has to go through an ethics committee, so you have the opportunity to catch research in human subjects and check whether or not it has reported the results, but one of the problems is that other forms of research are much more of a moving target.
Dr Kolstoe: I take sideways responsibility at my institution for research integrity and ethics. We have a policy that all research is ethically reviewed. There is quite a range of research. We go from a simple checkbox all the way up to requiring ethics committee review, but the advantage of that system is that we have a list of all the research that is going on and we can check. We often get criticism from our academics: “We don’t have to do this at such and such an institution. Why are you imposing this system on us?” Personally, I believe that having a system and a university that makes sure everything is ethically reviewed allows a certain amount of accountability that just gets missed in those other places.
Q339 Chair: Do you require the publication of all results?
Dr Kolstoe: We ask that of our researchers. There is still an argument as to exactly what publication means, and whether or not it is in a journal or by other means of dissemination.
Dr Goldacre: That shows the difficulty with non-trial research. With clinical trials, there are lists of exactly what information you would like to know about them, so it is very easy to say, “I don’t care if it is in a journal or on a website. This is what I want to know when a trial is finished.” When it comes to other forms of research, it is less clear what publishing your methods and results means, so that is another example where trials are the best testbed.
Q340 Chair: Can I ask about the Cameron initiative at the G7 summit in 2015? He said that “the UK will look to develop an international agreement that would see the publication of results of all clinical trials of vaccines for relevant diseases.” What Government action has there been since that commitment was made at the G7?
Dr Goldacre: As far as I am aware, none.
Q341 Chair: Is any international effort going on to stick to that agreement?
Dr Goldacre: After that, in 2015, the World Health Organisation issued a joint position statement on the rapid sharing of data and results in public health emergencies of international concern. That was pegged in particular to Ebola. Arising from that meeting, our team in Oxford did an audit of all of the treatment and prevention trials conducted around Ebola. The results are currently under review, but I can tell you that there are many trials of both treatments and vaccines for Ebola that have completed but not reported results, including I believe some from the UK. I am happy to check that and pass on specific cases, if it would be useful.
Q342 Chair: The commitment in 2015 has not been seen through into any tangible action.
Dr Goldacre: You cannot improve quality unless you are auditing and monitoring where you are. The fact that it falls to us in Oxford as an independent group to conduct this audit and that these audits can be quite hard to publish, because they can make journals uncomfortable, shows how far behind we are, and that mystifies me. People are not trying to bury us, but they are not pleased to hear from us.
Dr Kolstoe: In the last couple of years, the Health Research Authority has got a lot stricter with trials that are run, very strongly encouraging trials to register. I am doing a little piece of work with them at the moment looking at improved registration rates and whether or not trials are reporting their results. The HRA is doing it and I am helping them to come up with some data on registration and how many of those trials have results, but there is no funding for it and it is being done pretty much as a hobby because the HRA does not have it under its broad agreement.
Dr Goldacre: Simon did a fantastic audit of all the trials approved by his ethics committee. It was published in a journal. He reported the percentage of trials that published their results, but not the individual trials that did and did not report their results. My understanding of that—correct me if I am wrong—is that the Health Research Authority would not permit him to share that information.
Dr Kolstoe: A confidentiality agreement was signed with the sponsors. The Health Research Authority was able to use that information for audit purposes, but we were not allowed to publish it.
Chair: Unbelievable.
Dr Goldacre: If you instructed the HRA to share that information and that it has to be in the public domain, I think they would respond positively.
Q343 Vicky Ford: You say, “unbelievable,” Chair, but I would like to know why they did that. There may be a reason why they decided to do that that we do not understand.
Dr Goldacre: I think it is because under the existing framework the information is given to them confidentially and they would need external instruction from somebody like yourselves to say that it is okay to share that information. I am sure they would be happy to do so.
Dr Kolstoe: I do not think the HRA should be held culpable. There is an agreement between them and the companies and sponsors, and they have to honour it, because of the Data Protection Act.
Chair: That is understood.
Q344 Stephen Metcalfe: I think it is clear that all three of you believe—correct me if I am wrong—that the HRA should have the authority to mandate the publication of all clinical trial outcomes. Is that right?
Dr Goldacre: At the very least, they should audit and share where that obligation is breached. I do not know that they can have any statutory power to impose penalties on people without legislation.
Q345 Stephen Metcalfe: No; it is more a matter of principle. I think you made it clear that, if you put the right rules in place, people in the main will want to comply with them.
Síle Lane: Yes, that’s it exactly; clear, strong rules. The HRA is in a good position to monitor and audit with those rules because of the information it holds.
Dr Kolstoe: I published a paper a couple of years ago about whether UK research ethics committees can monitor this. Yes, they absolutely can. A couple of years later, it was, should they monitor it? That is really the discussion now. Personally, I think they should be monitoring it; others feel perhaps not.
Q346 Stephen Metcalfe: Can you explain the position of those who feel that we should not? What would their argument be? There is consensus among you, but please can you play devil’s advocate for a few moments?
Dr Kolstoe: Probably the biggest barrier for the HRA is that it is not in their remit and they are not funded to do this sort of work. Although they have the information—they have a huge database of every clinical trial that has taken place in the NHS, going back 10, 15 or so years—they do not have the funding to go back to the sponsors to ask, “Have you published it and/or checked?” I had a masters student who spent a year looking at 116 studies that came through one of my committees. It took an entire year to do that work. It was doable, but the HRA do not have that in their remit, or the funding to do it, at the moment.
The feedback I receive when I share this at national research ethics committee meetings is that research ethics committees are volunteers; there is only just enough time to review the documents at the one point in time before a study starts, and there are not the resources to have a wider remit.
Q347 Stephen Metcalfe: By resources, do you mean money or people and time?
Dr Kolstoe: People and time.
Q348 Stephen Metcalfe: That is perhaps not as easy to solve as the money issue.
Síle Lane: It doesn’t have to be expensive.
Dr Goldacre: It is very cheap for the HRA to do that kind of checking because they can instruct people that at the end of their project they have to submit an end-of-project report in which they say what the results are. The difficulty arises when you have to go out and search for it manually. You have to feel very confident that you are right, that you cannot find the results, in order publicly to assert that they are missing, but it is very quick and easy for the HRA to say, “Tell us now; we are the research regulator,” so it should cost nothing.
Síle Lane: I echo some of what Ben said. There are online automatic tracker tools that Ben and his team in Oxford are developing. They are pieces of software that automatically pull in information on clinical trials from different sources. They search automatically using algorithms for the results of those trials. Those tools could be adopted by Government bodies and funders, or other funders, to monitor whether the people they fund are fulfilling their obligations. The tools could be the way that the funders who joined the WHO statement fulfil their obligations to uncover and monitor publication rates in their back yards.
Dr Goldacre: This is very important. The WHO joint position statement, about five months ago, was written in very concrete terms. It said, “We commit within 12 months of signing to have a policy that says we register all trials and publish all results.” They also committed to setting up a framework for doing public audits of their compliance. It is really important that they are held to account for that, so come July 2018 we need to make sure that we go back and check.
As long as people make the results available, I will find a way for us to do all of these audits. My team in Oxford has built two different streams of audit tools. One is manual audit tools, which is what we did on Ebola, where you manually search for the results of all the trials that have been conducted. It is very laborious. It is quite fretful, because you feel anxious; if you have not found the results, are you right to say they have not been published? As Síle says, we have built automated tools, which is what we did for the European trials register and for ClinicalTrials.gov.
If anybody ever says that they do not have the administrative framework to do audits, or there is nobody to do audits for them, we are at their service. We will do audits of compliance with trial reporting for anybody who wants it. I do not say that because I would like our team to have revenue for it. It is very boring work, but we would be happy to do it. I would like to see a rich ecosystem of people offering to do audits of trial transparency, but it should not cost that much. In particular, the funder or regulator holds all the keys; they can say to the people who have taken £1 million from them to do a trial for MRC, NIHR, Wellcome or Cancer Research UK, “Show us the results of all the trials we have funded,” but, frankly, they should also say, “Before we give you any money for a new trial, tell us about the trials you have done in the past.”
Q349 Stephen Metcalfe: You mentioned the WHO model. Are there other countries that do this better than us, or are clearer in their expectations?
Dr Goldacre: The US has the FDA Amendments Act 2007. That still has loopholes. For example, it covers only drugs and some things done by NIH; it covers only trials on currently approved products; it covers only trials with at least one site in the US, and so on. None the less, it requires people to post results on ClinicalTrials.gov within 12 months. It has been incredibly slow to come to implementation. The final rule-making, which is the sort of thing that has to happen before it can be implemented, did not happen until last year, and the first trials to be fully covered by it will not finish until January next year, at which point we will have an audit tool to show who is and is not complying, to the best of our abilities. Nobody has a requirement that all trials must report their results, as far as I am aware, but it is a very straightforward thing to do with clinical trials.
Q350 Clive Lewis: Dr Goldacre, you touched earlier on Europe, and the European Union (Withdrawal) Bill is going on behind your head. Some of the new clinical trials regulation—the EU law which will come into force in 2018—will be missed from that Bill. Is this new legislation something the UK should adopt? Is it a panacea? Would it actually solve many of the problems you have raised about transparency?
Síle Lane: It comes into force in 2019. It has recently been delayed, because the register has not been built yet.
Clive Lewis: It will still be too late for the EU (Withdrawal) Bill.
Síle Lane: Exactly.
Dr Goldacre: It is important that we are not only compliant with that but that we require the results of all trials to be made publicly available. There is a UK plc issue. The UK’s share of the clinical trials market globally has been dropping year on year; it dropped about threefold over a decade. That is because we cannot compete on cost for clinical trials in low and middle-income countries. We cannot compete on cost with trials in eastern Europe, Brazil, Russia, India and China. However, we can compete on quality. Increasingly, people have become very sceptical about trials coming out of low and middle-income settings. Clinical trials are an information business. Medicine innovation is all about information, not the treatment, but the fact of whether or not it works. That is what we care about.
Q351 Chair: You are saying that we become the gold standard.
Dr Goldacre: We become the gold standard; we become the first country in the world to say that all trials should be reported properly. Who wants to have their eyes glaze over thinking about the different requirements in different countries for different kinds of treatments? Do they require you to report all pre-specified outcomes, or just something on some website somewhere? You make the UK the gold standard, where you say, “This is the place where, if a clinical trial is done, you can be guaranteed that it will be published correctly.”
Q352 Clive Lewis: Would you go further than the EU regulation that will come in 2019?
Dr Goldacre: I would go further than the EU regulation in the sense that I would say it should be all trials, not just trials of medicinal products or drugs and vaccines. It has to be trials of surgical interventions, devices, talking treatments and public health interventions. You make the UK the gold standard.
Q353 Chair: But just within the medical sphere, the health sphere.
Dr Goldacre: That is where I would start, because everything else is so complicated and messy. What we learn about trying to fix it with clinical trials will generalise to everything else, but we do not have a hope in hell of fixing the other problems if we cannot fix clinical trials. The first paper on the prevalence of publication bias in clinical trials was published over three and a half decades ago, and we still have not fixed it. We should not get ahead of ourselves; this is the thing we can fix. This is an implementation job. We know about the principles; we know that we should publish results.
Dr Kolstoe: Those arguments make a lot of sense. When you listen to them, you think, “Why would anyone disagree?” I have been in meetings with phase 1 clinical trials companies and other clinical companies where there are howls of protest as soon as those sorts of suggestions are made, with people saying, “We’ll just take our business abroad.” That has to be thought about. You will get pushback from the industry; it is certainly something I have seen.
Dr Goldacre: We have a tiny percentage of the global clinical trials market. We have more to gain by saying that trials in the UK are the gold standard you can trust. We have much more to gain from that than we do from racing to the bottom and having bad regulatory standards like everybody else.
Síle Lane: It is certainly the case that pharma companies headquartered or based in the UK are leading the world in thinking about this. GSK and Shire in particular are the companies that have done the hardest work in figuring out the practicalities and costs of increased transparency. They have gone ahead and started to implement it. Ben and I and the rest of the team recently did an audit of pharma companies’ policies on clinical trial transparency. GSK topped that audit. According to the trial tracker tools that Ben is building, right now only one company in the world, Shire, has a 100% publication rate. UK companies have been leading the way on this.
The UK has led discussions on clinical trial transparency right across the world. Certainly, in Brussels during discussion of the EU clinical trial regulation, the UK Government led those discussions. The NHS and the HRA were right in there lobbying for more transparency. UK-based organisations like ourselves in the campaign, the medical research charities and the medical royal colleges are part of the strong voices calling for more transparency. Given the hard work by those companies and the Government, if the UK does not get to benefit from all of that, it is a waste of time.
Q354 Chair: Simon, do you disagree or are you saying we should go into this with our eyes open?
Dr Kolstoe: I agree exactly with what these guys are saying, but I have definitely sat in meetings where people from the pharmaceutical industry have been complaining.
Q355 Vicky Ford: We do not expect private companies in other sectors to report inventions that did not work.
Dr Goldacre: But the problem is that inventions that do not work do not get used. The great challenge with medicine is that it is not immediately apparent to you whether or not a statin is working to prevent a heart attack in 10 years’ time. You can detect that only by running a robust, well‑designed clinical trial. That is why the information is so much more important. If you invent a jet that does not take off, nobody cares, but if you invent a drug that is third best in class, and you manage to present it as if it is the best in class, people will use it, to the detriment of patients and the health service.
Q356 Vicky Ford: If you are working for a private company and you have to write up the trial of something that did not work—I put it in layman’s terms—presumably you are still being paid by your company during that time, but if you are a university researcher, you may not be paid for your write-up time at the moment, under the current model. If it was something that would have a positive result, you have a huge incentive for your name and career because any upside in the way of spin-off might be yours, but if you have to spend your time, for which you may not be paid, to write up a negative result, do we need to change the way funding works, so that researchers get paid for write-up time?
Dr Goldacre: They already are.
Vicky Ford: Okay.
Dr Kolstoe: I have some personal experience of that. I worked with a big pharmaceutical company and a project was not going in the way they expected. They pulled the plug on it and did not expect anyone in the company to spend any more time on that project. As an academic, that was frustrating for me, because I had worked on it for a few years and I had results that I wanted to get published, but the company said that was no longer their priority and they moved on. That is an issue.
Síle Lane: You have put your finger on exactly where researchers are in this, and why a bit of a chasm is opening up between what industry and academics are doing. It is the case that clinical trial researchers are obliged under laws and ethical guidelines to publish the results, positive or negative. There are dozens of reasons why researchers are obliged to do it. Different levers then come into play, don’t they? You have put your finger on the human reasons as to exactly why researchers are not that motivated to do it; they want to move on to the next most interesting thing. Funders, ethics bodies and Governments say they care about this, and expect researchers to fulfil their obligations, so the Minister saying that is important.
Q357 Chair: A large part of this, as we have said, comes down to efficient use of public resources. We are potentially wasting money if we are not learning from the failures as well.
Dr Kolstoe: According to Iain Chalmers, 85% of research is wasted.
Dr Goldacre: Every time I fill out a very long and boring grant application, I describe what our dissemination plan is, and the writing-up of the project is costed in. When we were launching AllTrials, we got some pushback from industry. In particular, one chief exec said, “You are asking us to spend all this time writing up studies from the past. We want to talk about building the new drugs of the future.” I think that is misguided and misleading in various ways. In particular, the cost of running a clinical trial is absolutely enormous, partly because of some very stupid overregulation. That is an important discussion for another day, because it is another area where the UK could compete in the global clinical trials market.
Q358 Chair: There is scope for streamlining as well as doing it properly.
Dr Goldacre: Absolutely. There is disproportionate regulation. Every trial in the UK of every treatment, whether it is high risk or low risk, is regulated as if it was a dangerous, toxic new cancer drug. To run very simple trials in primary care, comparing one statin with another, when those statins are already taken by tens of millions of people around the world, you are subject to all the same regulations that would apply to a new and potentially dangerous drug, which is absurd.
Q359 Chair: I think you all heard the evidence of the two witnesses from the States. The US has a system of regulation that we do not have. Do you find their arguments for why we should have something similar here compelling?
Dr Kolstoe: I was at the world research integrity conference in Amsterdam. It was clear that the United States seems to be leading the way in research integrity issues, but there is a very strong focus on federally funded projects. Looking more broadly, I did not hear a lot about what they are doing about industry-funded projects. In this country, 60% of other research is done more broadly. Here we have the advantage that we have the Health Research Authority and the NHS. Because so much research is going on in the NHS, the Health Research Authority can get a much better grasp of what is going on than anything in the States. The United States has institutional review boards instead of research ethics committees, but they are very much based in institutions, whereas here, through the HRA, we have NHS research ethics committees that are in a fantastic position to monitor these things.
You had some slightly misleading evidence from someone else a bit earlier who referred to the research ethics system about 10 years ago, with local research ethics committees. The HRA has transformed the research ethics committee landscape in the last 10 years and is in a fantastic position to look at these problems and come up with the data, and to be the place where an audit can happen and do it across the board.
Q360 Vicky Ford: You think the research ethics committees are now in a very good place.
Dr Kolstoe: Certainly in terms of the information they gather. Whether it is the research ethics committee itself that does those audits, or whether the Health Research Authority does it on research ethics committee records is another point, but it is in a brilliant place in the system.
Dr Goldacre: At no cost, HRA could be instructed to require an end-of-study report from people, saying whether or not they have reported the results, and, if so, where. They could be required to share that line-by-line information, so that they are not anxious about Simon publishing anything more than a percentage figure, and then we can take over from there and do the audits, but they need to be instructed to share that information.
Chair: Thank you very much indeed for a fascinating session.