Science and Technology Committee
Oral evidence: Research integrity, HC 350
Tuesday 30 January 2018
Ordered by the House of Commons to be published on 30 January 2018.
Members present: Norman Lamb (Chair); Vicky Ford; Bill Grant; Stephen Metcalfe; Carol Monaghan; Damien Moore; Graham Stringer; Martin Whitfield.
Questions 361 - 531
Witnesses
I: Professor Sir Bernard Silverman, Chair of Trustees, UK Research Integrity Office; and James Parry, Chief Executive, UK Research Integrity Office.
II: Dr Tony Peatfield, Director of Corporate Affairs, Medical Research Council, and Chairman, RCUK Good Research Conduct Network; and Dr Steven Hill, Head of Research Policy, Higher Education Funding Council for England.
III: Professor Jonathan Montgomery, Chair, Health Research Authority.
Written evidence from witnesses:
– UK Research Integrity Office
– Higher Education Funding Council for England
Witnesses: Professor Sir Bernard Silverman and James Parry.
Q361 Chair: Welcome. It is very good to see you both. Would you like to introduce yourselves briefly?
Professor Silverman: I am Bernard Silverman. I am the chair of the UK Research Integrity Office.
James Parry: Good morning. I am James Parry. I am the chief executive of the UK Research Integrity Office.
Q362 Chair: Brilliant. All of us will ask you questions. We are under quite tight time constraints, so will you try to keep your answers as tight as possible? Do not feel that both of you have to answer everything.
The UKRIO describes itself as a provider of support and advice to the sector on matters of research integrity. Do you provide that support passively on request, or do you have a leadership role in proactively trying to drive change in practice?
Professor Silverman: Both. We initiate projects to provide generic advice and discuss ways in which we should do that. For example, one important event is our annual conference. We go to a lot of trouble to decide which topics people would like to know about and invite senior figures to speak about them. In that way, we inculcate a good culture. Of course, we also offer a confidential advisory and support service; James can give more details of its use. That provides what I would call reactive advice, so we are both active and passive.
Q363 Chair: Do you want to add anything, James?
James Parry: There is also a crossover. For example, we have benchmark publications—a code of practice for research and a procedure for the investigation of misconduct in research—which are out there for anyone to use. We also provide tailored support to help institutions to embed those into their systems and practices, both in the generic sense—for example, by establishing or revising a procedure for investigating alleged research fraud—and by helping with practical advice during investigations. There is a lot of crossover in those elements.
Q364 Chair: You are funded in part by universities. Sixty-five universities are subscribers.
Professor Silverman: Our only source of funding is the subscriptions that we get. Each of our subscribers pays a modest £2,600 a year. They include not just universities but other kinds of research institutes and so on. The majority are UK universities.
Q365 Chair: My sense, as an ex-lawyer, is that that provides something of a conflict of interest. There is no problem in providing advice and support, but if you are seeking to achieve higher standards and to play a leadership role does it not compromise you in demanding change, calling out bad practice, naming and shaming, or whatever may be necessary to drive a change in culture? Is that an issue?
Professor Silverman: I do not think so.
Q366 Chair: Why not?
Professor Silverman: We work hard to make sure that there are not conflicts of interest.
Q367 Chair: If your survival depends on payments by universities, that inevitably has an impact, does it not?
James Parry: Because pretty much everyone pays a flat-rate subscription, we are not beholden to any one institution. If a university says, “We don’t like the way you are handling this particular case,” we can simply say, “That is all very well and good. We are happy for you to unsubscribe.” We do not tell our subscribers whom we are helping or how we are helping them. If someone comes to us from a university that is also a subscriber, we will declare to that person, “Just so you know, these people are one of our many funders. We will not share any confidential information with your institution about your inquiry.” The institutions know that and do not press.
Q368 Chair: You are described as the Research Integrity Office. Do you have any official status?
Professor Silverman: I do not think so. That is the title. We are probably called that for historic reasons.
James Parry: Yes. It mimics the titles of institutions in other countries.
Q369 Chair: I think that I know the answer to this question from your earlier answer. What support, financial or otherwise, do you currently receive from Government for your work?
Professor Silverman: None. We are independent of Government in that respect.
Q370 Chair: It is interesting that you are independent of Government, but not independent of the institutions that are doing the work that might be subject to concern.
Professor Silverman: If Government would care to give us a large subvention, we would not have to charge subscribers.
Q371 Chair: That begs a further question. Do you think that they should, so that you would be independent of the institutions that are doing the work that you are monitoring and where you are seeking to improve standards?
Professor Silverman: I would prefer to have a diversified source of funding. The levels of funding are so modest that I do not think that we should worry too much about it. My main role, as chair of the trustees, is to ensure the financial stability of the charity, of course. I am happy at the moment, because we have a lot of different subscribers. The numbers are going up all the time. We are diversifying in international coverage and so on. I do not worry where the money comes from, as long as we are transparent about it.
Q372 Chair: You would not suggest that Government should provide funding support, to enable you to be independent.
Professor Silverman: We would be delighted, because then we would not have to spend time collecting subscriptions and so on. If Government would care to give us a subvention, we would be delighted. Of course, there is always the risk that they would take it away again. In a way, having a subscriber base gives longer-term stability. We do not see this as a big issue.
Q373 Vicky Ford: All that I wanted to hear was whether it was an issue.
Professor Silverman: No.
Q374 Chair: Do you wish to comment, James?
James Parry: Bernard has dealt with the point very well.
Q375 Bill Grant: My question is about how you measure your performance. How do you establish whether you are doing a good job? Are you subject to any formal evaluation by an external organisation? Has anybody looked at what you are doing? Has anybody measured your performance—hopefully, with a good outcome?
Professor Silverman: I do not think that we have had an external measure of our performance. Would you like to tell us about that, James?
James Parry: During our pilot phase, from 2006 to 2010, we had external evaluations, because at that stage we were hosted by Universities UK and funded by the higher education funding councils, the Royal Society and other bodies. The pilot was assessed and found to be a success. At that point, we spun out, became operationally independent from Universities UK in legal governance and became a charity.
We have not had external evaluations since then. The metrics we look at are growth in the use of our advisory service, which has happened year on year, growth in the number of subscribers, which has increased every year, and growth in the number of proactive approaches by institutions to try to get things right, which I think is most important. I refer not to crisis management, but to institutions asking, “How do we make sure that this particular project or type of research goes well? How do we make sure that, if we have to investigate anything, it runs smoothly? How do we make sure that our systems for ensuring integrity and reviewing research ethics and research governance go well?” That category of request for support has grown every year, as has the number of requests for training and development.
Q376 Bill Grant: Are you using the expansion and growth in the number of subscribers as a measure of success and as a good indicator?
James Parry: It is a measure—personally, I do not think that it is the most important measure. For our financial stability, it is useful. It shows that institutions are more willing to engage with us, because we give more long-term, in-depth support. I would say that the main metrics are the increasing use of the advisory service and, in particular, the number of proactive requests for support by institutions. That has grown as well.
Q377 Bill Grant: Are you indicating that you are evolving to become a trusted organisation?
James Parry: I would like to think so. Quite frankly, when we moved to the subscription model of funding in 2010—Bernard was not there at the time—it was a bit of a leap into the unknown. We had never done it before. Within two years, we had 24 universities signed up. Two years later, it was just under 50, from memory. Now we have 79, including non-universities and universities outside the UK. Therefore, there is an element of being a trusted brand.
Q378 Bill Grant: That is looking to external activities. Do you have any internal targets or measures you report against? Do you have any internal reporting?
James Parry: Yes. We have our internal appraisals and the like.
Professor Silverman: The chief executive reports to me. The board of trustees and I are the people to whom internal reporting is done. What I am particularly pleased about as regards performance is the diversity. I must clarify that one of our subscribers is a Government organisation. Is that right, James?
James Parry: Yes—the Ministry of Defence Research Ethics Committee.
Professor Silverman: The Ministry of Defence Research Ethics Committee is one of our subscribers. I apologise for not making that clear earlier.
Q379 Bill Grant: The majority of UK universities are not subscribers to your organisation. I sense that you must find that disappointing. Are you minded to change that? If so, what efforts will you make to change it? Finally, what reasons do those who do not subscribe suggest to you for not wanting to engage with you? That may be quite a complex question.
Professor Silverman: One does not like to distinguish all the time, but the majority of strong research universities are members. Only a very small number are not. Those are the ones we make proactive efforts to recruit.
Q380 Chair: Are all Russell Group universities members?
James Parry: Twenty out of the 24 are members.
Professor Silverman: The vast majority of the Russell Group are members. If we had a request for help from a non-member university, we would encourage it to become a subscriber. We do not restrict our advice to subscribers, but we encourage universities to become subscribers. Our numbers are growing all the time. One of James’s targets each year from me is to increase the number of subscribers. He more than meets that each time.
Q381 Bill Grant: Having captured the subscribers, have you lost any? Has any university failed to renew its subscription, and given you a reason for why it has not renewed?
James Parry: Over the years, three dropped out, citing financial constraints. Two of those three have since re-subscribed, one in less than a year and one about two years later.
Q382 Bill Grant: You can take some heart from that. They left and returned.
James Parry: Yes. Often the people doing the practical work on issues of research integrity in universities are not the people who make decisions about whether a university makes a corporate subscription. They are going to other budget holders and making a request.
Q383 Bill Grant: Finally, I have one question on an important aspect of research. The Research Integrity Office provides model research misconduct processes. How many universities use those model processes?
James Parry: It has been a few years since we did a survey. A few years ago, we surveyed 75 institutions. Over 50 were making use in some way of our research misconduct procedure and our code of practice for research, which are our core publications. Those have also been endorsed by funding bodies. They do not necessarily mandate their adoption, but they recommend them. Last year, for example, Research Councils UK said that a couple of provisions in our research misconduct procedure—for example, having an external member on a formal investigation panel for research misconduct—must be adopted by institutions as a condition of their grants.
Q384 Chair: In your view, how many of your subscriber organisations meet the requirement to have an external member? Do you think that it is uniform among your subscriber members?
James Parry: I think that the vast majority meet it. Both of these documents first came out about 10 years ago. Since they were published, we have worked with institutions to adapt them and to embed them in their procedures. For example, we always recommend that, if you have a formal investigation panel, you should have at least one external member on it. You should also have the option of the chair of the panel being an external member—for example, if the university is dealing with an allegation concerning research that has been the subject of much public debate and concern. We also recommend that, in complex inquiries, you should have provision for getting an external member in the screening stage, too.
Professor Silverman: We cannot monitor every single inquiry to discover whether they are using our procedures, but by being a trusted organisation and having published procedures we hope that we give a model for the institutions themselves.
Q385 Vicky Ford: At previous hearings, one of the reasons given for why research is sometimes misleading was that sometimes the researchers themselves do not have a clear enough understanding of statistics. Do you agree? Could you help to provide training in that? How do we embed that understanding of statistics in our scientists?
Professor Silverman: As you may know, I am a statistician myself.
Q386 Vicky Ford: That may be why I asked the question.
Professor Silverman: I read the transcripts of those sessions. I thought that what was said there—particularly by David Hand, who is a very old and close friend of mine—was absolutely to the point. I do not think that it is the job of the UK Research Integrity Office to provide training in statistics. I would look to the Royal Statistical Society and places like that.
Q387 Chair: Do you think that there is a need for it?
Professor Silverman: In my capacity as a former president of the Royal Statistical Society, yes, I do, but I do not think that it is particularly a matter for the Research Integrity Office. In today’s capacity, I would not offer an opinion.
Q388 Vicky Ford: But you would not disagree with them.
Professor Silverman: I would not disagree with Professor Hand. I am speaking in a personal capacity, not on behalf of the UKRIO.
Q389 Carol Monaghan: I am slightly concerned that, because an institution does not subscribe to the UKRIO, its research is somehow downgraded and not considered to be as solid and acceptable as that of other places. Is there a danger of that? If so, are institutions almost being held to ransom over joining?
James Parry: While we are a trusted brand, I do not think that we are that much of a trusted brand. Funders and publishers neither mandate that institutions subscribe to us nor check whether an institution is a subscriber when research grants are applied for and research is submitted to publishers for publication. I do not think that there is a two-tier system in that sense. I would like to think that we are seen as worth subscribing to because we provide practical help, not because, if you do not, you will be considered somehow substandard.
Q390 Carol Monaghan: So you are a carrot, rather than a stick.
James Parry: I would like to think that. When it comes to research integrity and research culture in general, it is easier to build sticks into systems, rather than carrots. We should look more at carrots in general terms.
Professor Silverman: We publish our list of subscribers, don’t we?
James Parry: On our website.
Professor Silverman: If you wanted to use membership of our organisation as a stick, you could. That would please the trustees, in the sense that our financial situation would be better. To be serious, we would not be very pleased to be used in that way.
Q391 Carol Monaghan: UKRIO has highlighted to us that it about to embark on a research project on research integrity. Will you tell us a bit about the timescale and methodology for that and the reasons behind it? Do you think that any recommendations will be made off the back of it?
Professor Silverman: It is a fairly small-scale project. It is research into research misconduct, because we feel that there is not a clear enough picture of what research misconduct there actually is. We know what types of things one should worry about, but it would be very interesting to get an up-to-date picture of how much of each problem there is.
It is a fairly small-scale project, using existing sources of information that are either in the public domain or can be obtained by making inquiries, if necessary, through FOIs, to universities and so on. We reckon that it will take about six months. It will be funded by ourselves. We are scoping the exact details at the moment. Our output will be a high-level summary and discussion of conclusions. Mark Walport says that there are three main areas of concern: research misconduct, questionable practices and sloppiness. It will be interesting to see how much of each of those things there is.
You ask about recommendations. We are not seeking evidence for recommendations that we have already formulated, but we intend that recommendations will come from this. When we have the research, we will look at it and make recommendations.
Q392 Carol Monaghan: Can we expect the recommendations in the next six months, or will they be on a longer timescale?
Professor Silverman: They will be made when the research is done; I would rather do it in that way. I think that we will be able to move to that quite quickly. It will also help to shape our own future work.
Q393 Chair: Do you go into this project with any view about the scale of the problem?
Professor Silverman: The evidence is that the problem is not vast. There is not a big problem, but the issue is important and, therefore, always needs attention. In other words, as Mark Walport says, there is a small amount of genuine research misconduct. There will be a larger shadow of sloppiness and so on. I do not think that there is any more of a problem than there ever has been. I am very glad that the research community is taking issues of integrity seriously.
Q394 Graham Stringer: That is an assertion without any evidence, isn’t it?
Professor Silverman: Retraction Watch had some interesting evidence in a previous session. It looks at the number of retractions per billion dollars spent on research. The absolute level is quite low. That is within a culture where we are now much more likely to retract bad research than we were in the past. That is a piece of evidence.
I do not have numerical evidence about sloppiness, because it is a very difficult thing to quantify, but I read the journals in my own field and others. In general, the papers that you read are produced to a good standard, but inevitably there are some that are not like that. But you are quite right, Mr Stringer; I agree with you.
Q395 Chair: There is a POSTnote on this whole issue. It refers to a 2009 study in which researchers were asked for their private view on the issues. It says that “2% said they had falsified, fabricated or altered data”—this is international, incidentally—and that “34% admitted” some “questionable research practices.” That is quite significant.
Professor Silverman: Yes, it is. Of course, some “questionable research practices” may be once they had done something that they subsequently regretted—but, yes, I agree.
Q396 Chair: The slight worry is that university after university returns nil inquiries each year. There is a sense, if there is any truth in that, that there is an issue that remains covered up—that the misconduct and questionable practices that are happening are not getting exposed. Is that your view?
James Parry: The 2009 study you are referring to was by Daniele Fanelli, who was then at the University of Edinburgh. The meta-analysis was based on existing studies. One of the reasons we want to do this piece of research is that a lot of those studies are either high-level international ones or based on data from the United States, so we are not sure that we have a true national [i.e. UK] picture. I definitely agree that the numbers quoted by Fanelli are too high and that things need to be done.
If an institution reported year after year that it never had any allegations, I would be somewhat sceptical about whether that figure was accurate. At a meeting of the Royal Society a couple of years ago, a panel I was on was asked, “What is the right number of allegations for an institution to have per year?” The answer is, “However many are honestly reported.” If the number is zero consistently, you may need to look at your practices and overall research culture.
What I also find interesting about the Fanelli study is the level of questionable research practices. It is vital that institutions—and funders and publishers; it is not just the universities—address research misconduct robustly, fairly and as transparently as they can, under employment law.
Q397 Chair: Do you think that more needs to be done?
James Parry: I think that more needs to be done by everybody. Based on the inquiries that come to us and studies like Fanelli’s in 2009, questionable research practices are an area of concern, because they are more common and, perhaps, more insidious.
It is worth noting that the research integrity concordat asks for data on “formal investigations” in the annual statements. That does not mean investigation that has been undertaken under a formal process—it means the second stage of an investigation, where they have assessed an issue, to see whether there is a case to answer, and decided that there is some evidence and there may have been misconduct. The issue then goes to a formal panel. Historically, that has been the level of information that has been asked for by funders, although last year Research Councils UK changed it. It said that—along with the Wellcome Trust, I believe—it wants to know about allegations that are being screened. It will get better reporting in that way.
Professor Silverman: Every university ought to publish research integrity statements that are easily accessible. We have tried to find all of them, and not all universities do it. It would be good if all universities did that.
Q398 Chair: And it should be a requirement.
Professor Silverman: We cannot make—
Q399 Chair: Whether you can or cannot, do you think that it should be?
Professor Silverman: Speaking personally, yes, I do.
Q400 Carol Monaghan: Would you consider publishing an annual report on the state of research integrity? What might that cover?
James Parry: As a charity and advisory body, we can publish the data on the concerns that people bring to us. We would be wary of saying that that is utterly reflective of the UK, because there are people who may decide not to access our support. We can certainly publish that sort of information. I have talked about institutions using our services, but just under 50% of our work is with individual researchers and members of the public. We want to make sure that they feel that they can come to us and that their concerns are confidential. We would not want to release data where someone could say, “Oh, that must be about my inquiry,” but we can certainly look at publishing a degree of information.
Professor Silverman: The research project that we are doing now will be a model for the kinds of things that we should do. We will take that one away. Obviously, one would produce summary data, not individual data. We would not say, “The University of X did this, that and the other.” However, that is a very good recommendation. It is one that we will take away.
Q401 Carol Monaghan: Do you feel that the frequency with which your advice has been sought is increasing? Are you collecting data on that to measure it?
James Parry: The use of our services is growing year on year. Like other things, such as the rate of retractions, that may be simply because there is more awareness of these issues and of us. For example, after the publication of the concordat in 2012, we got a lot more attention, because we were name-checked in that document and because it brought research integrity afresh to those at a very high level of institutions.
Professor Silverman: I do not think that there is more of a problem, but we are delighted that our services are being used more.
Q402 Carol Monaghan: It is more about awareness.
Professor Silverman: Indeed.
Q403 Carol Monaghan: Does the word “pauchling” mean anything to you? Maybe that is a Scottish word.
Chair: You have flummoxed us all.
Carol Monaghan: I am glad we have got that on the parliamentary record. “Pauchling” is cooking the books, often in a scientific manner.
You cover all sections of research integrity, including the private sector. Do you detect any differences in the approach or attitude towards research integrity—or, possibly, the frequency of problems—between the two sectors? Where do you think that most of your work is required?
James Parry: We cover any research that is done under the auspices of the UK. That includes research in any sector and discipline, whether it is done by an individual or by an organisation. Historically, the private sector has been a relatively low user of our services.
Q404 Chair: Do no private sector organisations subscribe to you?
James Parry: We have one private sector subscriber—the RAND Corporation, which joined recently. We do not have any other private sector subscribers. However, we hear from the private sector on some occasions. When institutions and individuals from the university sector come to us, it may be about a project where they are partnering with the national health service, the private sector or a charity.
Q405 Carol Monaghan: Does it concern you that the private sector does not engage to the same extent as the academic sector?
James Parry: We would welcome it if it engaged more.
Q406 Carol Monaghan: But it has not happened so far.
James Parry: That is correct.
Q407 Martin Whitfield: May we explore the concordat a little more? You say that it was signed in 2012 and that you were name-checked in it, but you are not actually a signatory to it, are you?
James Parry: That is correct. We wanted to maintain our independence.
Q408 Martin Whitfield: That is to maintain your independence from the concordat. Do you press or advocate the concordat? Do you inquire of your signatories and other users about their involvement with the concordat, or does it sit separately from you?
Professor Silverman: The concordat shapes all our activities. Everything that we do is within the principles of the concordat. We want to help it to work out and to do things that will support it. The matter that I just mentioned—publishing research integrity statements—is an example of something that I would press a little, perhaps even through this inquiry.
Q409 Martin Whitfield: For all intents and purposes, you advocate all the views of the concordat as being a positive way forward, both for you as an organisation and your signatories, and for everyone who is in the field of research, across the board. Whose responsibility do you think it is that the concordat is implemented? Where does the buck stop?
Professor Silverman: Within the concordat, there are places where the buck stops. It places obligations on institutions, funders and so on. What we have here is a complicated system with lots of moving parts. The concordat provides a framework for that.
Q410 Martin Whitfield: Who do you think should take responsibility for the concordat to be implemented? I refer not just to its terms, but to making sure that people who are signed up to it—
Professor Silverman: Do you mean the overview of the concordat?
Martin Whitfield: Yes.
Professor Silverman: At the moment, it is like morality and ethics in general, I suppose. There is no overarching body. Is that fair?
James Parry: As is the case in every country, the system relies on self-regulation by researchers. On top of that, you have the universities’ responsibility to make sure that researchers are aware of their responsibility to self-regulate, to support them in doing it and to take action if they do not. The funders are there to monitor the universities’ progress.
We see our role as being to help institutions and individuals to implement the concordat and to shy away from trying to tick the boxes for the sake of contractual compliance. A lot of work that we do with institutions and individuals is to help them to dial down this document, which is quite high level and overarching, and is meant to cover all types of research in the UK, so that they can make practical changes, using the document and other good-practice examples, to make things better in their institution. For example, we have a self-assessment tool for the concordat, which includes guidance on how to write an annual statement. We are doing another guidance note on that topic, which should be out later this year, hopefully.
Q411 Martin Whitfield: I am sorry to interrupt you; I would like to come back to your toolkit in a moment. Do you think that you could take the moral high ground and be the overseers of the concordat’s implementation? It seems to fit very much with your ethos of independence and being removed from the universities, as we discussed in relation to funding, and you recognise its importance. I am not asking you to make a decision today.
Professor Silverman: We see ourselves as overseeing it informally, in the sense that it shapes our activities. I do not think that we would seek a formal status, because that would make it harder for us to be an adviser and a supporter at the same time. It is about being a formal regulator—that is the word that we use—which would not sit comfortably with the way in which we like to work.
Q412 Martin Whitfield: But you are content to continue to advocate the importance of the concordat.
Professor Silverman: Yes. Although we would not necessarily advocate there being a formal regulator, we would work within whatever system there was.
Q413 Martin Whitfield: We have discussed the potential for an annual report. Do you see your role extending to an annual review of where the concordat sits within that in relation to research within the UK?
Professor Silverman: The concordat would be the jumping-off point for any annual report.
Q414 Martin Whitfield: That is useful. You mentioned the self-assessment toolkit. It is used by your funders—by the signatories to it—but it is not available to everyone.
Professor Silverman: It is.
James Parry: We made it available recently.
Professor Silverman: It is now available to everyone.
Q415 Martin Whitfield: So, as well as having an offer of annual oversight, we have the tool to ascertain how well the concordat is being followed.
James Parry: It is a tool. I should point out that it is not endorsed by the funders. We developed it separately, looking at the themes that we feel cut across the concordat’s commitments and offering suggestions on how you can translate them into practical terms.
Martin Whitfield: Effectively, you are acting as the interface between the concordat and its users. You are prepared—quite sensibly, because you are where the knowledge rests—to give advice, understanding and implementation to it. That is very positive. Thank you.
Q416 Chair: It has been in place for some years now. You have identified one area—annual reporting, as we have described—where you think that it could be tightened. Do you think that it is time for it to be refreshed? You described it as “high level.” Do you think that it needs to be clearer in any areas about the universities’ obligations? It includes recommendations about ways in which it can be complied with. Do we need to look at it again?
James Parry: As you will be aware, Universities UK did a review of the concordat in 2016. We canvassed our subscribers extensively and then presented anonymised views. One thing that came out was that the concordat is designed to promote research integrity but is also an element of contractual compliance with research funders. Therefore, slightly improved clarity from funders on what their expectations are and how they are monitoring them would be helpful. It is good that Research Councils UK, for example, publishes a summary of its compliance data from the institutions. It would be useful to look at those datasets from other funders as well.
The concordat approach has garnered a lot of international interest. The Republic of Ireland set up its own research integrity concordat model based on the UK system. Our international subscribers, which include institutions from Ireland and elsewhere, have found our self-assessment tool for the concordat useful as a way of measuring their own research integrity provision. There are many routes that a country can take to ensure integrity in its research. The UK’s is getting a lot of attention from other countries.
Q417 Chair: The point is not to argue that it is not the right approach, but to argue whether it can be improved, essentially.
James Parry: Yes. Every country has adopted slightly different variations. Some have gone down the very heavy regulatory route, but often those regulators regulate only certain elements of research integrity, certain funding streams or certain disciplines. Others, such as the UK, have an advisory approach. In Germany, the organisation is an ombudsman that regulates disputes between academics. Serious fraud is investigated by the institution. There are a variety of approaches out there.
In the global research integrity community—that is probably giving it a grander title than it deserves—everyone is very keen to learn from everyone else. I have found that to be the case within the UK as well. A lot of what we do is sharing good practice. If an institution comes to us for advice, we will give advice as best we can, but we will also say, “We are aware of other institutions that have grappled with this issue. We will put you in contact with them.” There has not had to be regulation to make sure that institutions talk to one another, share lessons learned and work together to improve things.
Professor Silverman: I come back to the question. We do not think that it is necessary to revisit the concordat itself. It is more important to continue to make sure that it is embedded, in the ways we have discussed.
Q418 Damien Moore: Good morning. You touched on other countries in your last answer. What assessment have you made of the merits of other countries’ models, such as those of Denmark, Australia and the United States, for regulating and responding to research misconduct? You have mentioned a couple of countries, but have you looked at misconduct regulation there?
Professor Silverman: Most of the regulation in other countries is regulation of investigations. I have looked at the American model. One thing that we have to understand is that culture is different in different places. The American model can be a bit legalistic. There are very strong sanctions within the American model, but it is mostly about regulation of the way in which formal investigations are carried out, once they take place. Some aspects of different international ways of working are relevant to us, and others are not.
In general, the larger issue of things like how to encourage a good research culture, training and how to deal with things informally is not specifically regulated elsewhere. That is the vast majority of the work in this area, anyway.
Q419 Damien Moore: In your view, should there be a change in the processes and structures for tackling research misconduct, through external oversight or regulation?
Professor Silverman: I do not think so. Our view is that it is the responsibility of employers. In the end, I suppose, the legal responsibility rests with the employer. What we do is provide templates of good practice for universities and other research employers to use in the unfortunate event that there is a formal investigation.
Q420 Damien Moore: What are the pros and cons from a legislative perspective of setting up an independent body to handle this instead?
Professor Silverman: I am not a lawyer. My concern with it is that it becomes an excessively bureaucratic process. We are agnostic, I suppose, on the way in which serious investigations are dealt with, but I think that it is not straightforward to set up such a body. If the decision were made to do so, we would, of course, be ready and willing to give advice on how we felt it could best be done. I think that there are a lot of difficulties in setting up a formal body of that kind, but we do not have a strong view on that.
Q421 Damien Moore: Do you find that a whistleblower would turn to you? Have they turned to you, if they are not happy with a university’s investigation process?
James Parry: Yes. We give advice to people with concerns about research misconduct, to people who are witnesses in cases and to respondents. We have found that we can effect change. I remember that in our very early years, 2006 to 2007, we found that institutions that were hesitant to take action—these days it is very different—could be prompted by a complainant writing back to them saying, “I think that you should look into this matter. I have contacted the UK Research Integrity Office, anonymously and in confidence, and am responding to advice received.” The idea is that a knowing third party prompted people.
These days institutions and publishers—which deal with allegations as well—recognise that it is part of good governance to deal with these matters properly. They cannot keep these things behind closed doors any more, as they may have wanted to do 10, 20, 30 or 40 years ago. Things get into the public domain. Both good practice and a degree of enlightened self-interest are driving this. People think, “If we don’t deal with this properly, it will come out sooner or later.”
Q422 Damien Moore: You never find that there is a conflict of interest, considering that universities are your main funders. This is a decision that you take.
James Parry: I go back to what was said earlier. If someone within our organisation had a personal conflict of interest, obviously we would not involve them. For example, Bernard had a position at Oxford, so we would not involve him in a matter relating to Oxford. However, we simply do not tell the funders—the people who subscribe to us—about inquiries relating to their own institution. We keep that confidential. Given the way in which our funding model is set up, if any funder ever kicked off, we could simply say, “Fair enough. You do not have to subscribe to us any more.”
Q423 Damien Moore: If the Government were to introduce external regulation in some form, would you see the future for your institution being the same as it is now, or would you question your own existence?
Professor Silverman: We would probably have a stronger future, because we would be providing advice within that structure. That is why we are agnostic, in a sense. We do not think that we should be the people formally to supervise things; we hope that the universities will do it for themselves. If Government felt that that should be the way, we would work with that, too.
James Parry: We have a body of evidence to inform the creation of a regulator, if that is felt to be desirable.
Q424 Graham Stringer: Is not the real regulator of science replication? The problem is that we do not know what is going on because people are trying to replicate science less and less. That has become a culture. People want to break ground, to get high impact in journals. They do not do what is really the slogging work of science, which is to try to repeat the experiments.
Professor Silverman: That is interesting. In my view, it is actually the other way around. My view is that we now have so much research going on that, within every project, you are replicating a lot of what has gone on before. In my own field, it is now considered standard that, if you produce a new statistical technique or work on some data, you make your technique publicly available in detail, so that your computer programme and so on is all in the public domain. You also put your data in the public domain, as far as you can.
When I started in research 40 years ago, that was not the general practice at all. Especially with the internet—the web—it becomes easier to replicate things. I completely agree about replication and continuously revisiting things being the way in which science is actually regulated, but I do not think that there is less of it than there used to be; in fact, I think there is more. One of the reasons why Retraction Watch has seen more retractions may be that replicability is now more accessible. This is a matter of opinion that we could discuss for a long time.
Q425 Graham Stringer: That is interesting. Staying with the culture of research, a lot of the written evidence that has come in has said that the problem of research is the culture of science—that scientists are looking for high impact and have short-term funding. Putting fraud to one side, where there are mistakes and shoddiness, are not you dealing with the symptoms, rather than the causes?
Professor Silverman: There is undoubtedly a very competitive culture. Actually, there always has been. There are cases 200 years ago of people stealing one another’s work because of the competitive culture. There is tension between pursuing a co-operative enterprise of trying to uncover truth, knowledge and so on, and doing so in a competitive way. I do not think that that necessarily leads to sloppy practices, because it means that people are always looking over one another’s shoulders.
Are we addressing the causes rather than the symptoms? The underlying cause is that there is always a small number of people who, unfortunately, will try to get around the system and do things in the wrong way. That is true in every aspect of human life. We have to understand that. We are doing our best to encourage people not to be like that.
Q426 Graham Stringer: We have HEFCE and Research Councils UK next. Do you think that they are doing enough to improve the culture and integrity of science?
Professor Silverman: They are very—
Q427 Graham Stringer: What questions would you ask them? Can you do our job for us?
Professor Silverman: Yes; exactly. I would ask RCUK and HEFCE whether they feel that they are expecting students and researchers to be sufficiently informed and to put integrity sufficiently at the front of their minds. Is there more that could be done to ensure, for example, that people who do PhDs have proper training in this area?
That is one thing. I said earlier that maybe HEFCE should encourage or require universities to publish research integrity statements prominently. Maybe it should audit them to make sure that they are not just lip service.
I believe that HEFCE and RCUK take this seriously. In constructing the new REF guidelines, they have tried to produce guidelines that genuinely encourage co-operative research and the assessment of institutions and groups, rather than just individuals, and that ask people to document impact and so on. Those are all positive steps in this regard.
Q428 Stephen Metcalfe: You mentioned PhD training. Two questions spring from that. First, do you think that that should be formal training, with formal recognition—a qualification to research, if you like? Secondly, I go back to the point about statistics. Do all research projects use statistics? If so, should that form part of the formal qualification to conduct a PhD research project?
Professor Silverman: First, do I think that there should be a formal qualification for PhD students in research integrity? This is not the considered view of the UKRIO, but I think that it would be a good thing to work towards. It might need to be tailored somewhat to the subject area and might be a matter of accrediting training that universities are providing, but I would say that it would be a good thing. After all, a PhD is not just doing research—it is a training in research. I think that some formal work on research integrity would be very worth while, just as university lecturers nowadays get formal training in teaching. I had that when I became a university lecturer, but I can tell you that not all my colleagues did.
You ask about formal training in statistics. Again, I am now speaking personally. I think that it would be difficult to require everybody to have formal training in statistics. I suggest that you go back to some of the witnesses in your earlier sessions to ask them about that. Of course, what can I say? The more training in statistics there is, the better.
James Parry: The question of training is important, potentially, at all career stages. Sometimes the basic standards for research are perceived as quite obvious, but the calls to our advisory service and other issues suggest that things are perhaps more challenging to put into practice. The question is, are we teaching researchers what they need to know, or are they picking things up as they go along? If it is the latter, are they picking up good habits or bad?
Authorship in academic publications is a good example. Some disciplines have formalised authorship criteria. In others, it is more a matter of custom and tradition: this is the way in which it is done. But both types of disciplines still have problems with authorship disputes. Those are insidious, because they can damage research teams. Authorship disputes can also stop papers being published, so this is an example of a questionable practice having quite serious consequences. An element of research integrity and ethics training interleaved throughout career progression as a researcher—not just for PhD students—could be very helpful.
Q429 Stephen Metcalfe: The question was about whether it should be formally recognised, as a licence to research, so that you could point to it and, potentially, remove it at some point.
Professor Silverman: I think that removing it would be difficult.
Q430 Chair: Do you think that someone who is found out—someone who has failed to meet the high standards of integrity in research—should continue to receive funding for further research?
Professor Silverman: If someone is a truly egregious offender, they should not, but how you enforce that is difficult.
Q431 Chair: We have identified that deliberate misconduct is a small but very serious issue. We know that it is there. We know that there is a bigger issue of sloppy standards and so forth. You have said very clearly that more needs to be done. We all need to do more. In a way, you are in the perfect position, given the work that you do, to make a judgment about what needs to be done. In guiding us, you will shape the recommendations in our report. You have talked about the need for better training and, perhaps, making annual reporting mandatory. What other clear recommendations do you have? Rather than just saying, “We need to do more,” can you give us your specific prescriptions for what needs to be done?
James Parry: There is a lot that needs to be looked at in terms of culture and leadership. It is important that we do not micromanage researchers, straitjacket them with bureaucracy or treat them all as potential research criminals. A compliance culture gets you only so far. Training and development through a researcher’s career progression, from PhD student to veteran professor, is very important.
We also need to look at how universities, publishers and every other actor in the system are contributing to the broader issues of research culture, and how we can make sure that researchers feel that they can put their hands up and say, “I have got a problem. I am not sure what to do.” To me, the most important line in the concordat is the one that says that there should be no stigma attached to researchers who find themselves in need of help. Institutions—in which I include publishers, regulators and funders—should defend researchers who stand up for research integrity in difficult circumstances. For me, those are the two most important bits of the concordat. That is about culture—the climate within a research team, a faculty or a school.
Q432 Chair: Sir Bernard, do you have any additional points?
Professor Silverman: I do not. The difficulty in trying to steer culture is what levers you can use to do that. I do not have any specific suggestions of particular levers, other than those that we have already made. I am encouraged by the fact that there is now a general understanding that this is an important issue. I hope that the work that we do will continue to push in that direction.
James Parry: We are working with the Royal Society on issues of research culture. There will be some outputs later this year. We will certainly keep the Committee informed of that work.
Professor Silverman: We are very pleased to work with the Royal Society. That is an example of our working with a highly respected and influential organisation. For this to be high on the agenda on the Royal Society was quite a win. That is the sort of thing we are very pleased with.
Q433 Chair: Thank you very much for your time. It is enormously appreciated.
Professor Silverman: Thank you very much.
Witnesses: Dr Tony Peatfield and Dr Steven Hill.
Q434 Chair: Welcome, both of you. Do you want to introduce yourselves quickly?
Dr Hill: Good morning. I am Steven Hill. I am head of research policy at the Higher Education Funding Council for England. As the Committee will be aware, the council’s research and knowledge exchange functions will transfer to a new organisation, Research England, in April. I will be in a similar role in Research England in the future.
Dr Peatfield: I am Tony Peatfield. I am director of corporate affairs at the Medical Research Council, as my day job. I am here today mainly because I also chair the RCUK good research conduct network, which is the body that joins together the research councils in the area of good research conduct.
Q435 Chair: I have a question for both of you. How would you describe your roles in relation to research integrity as funders? Where do you fit into the research integrity system as a whole?
Dr Hill: We are a substantial funder of research in universities in England. Our role around research integrity is to provide incentives and frameworks within which autonomous universities can deal with matters of research integrity. The primary route by which we do that at the moment is related to the concordat you have already talked about this morning. We make compliance with the concordat a condition of our grants. All universities that receive research funding from us are required to comply with the concordat.
Q436 Chair: We have heard that the concordat, welcome though it is, is quite high level and leaves quite a lot of scope for interpretation. Do you think that compliance with the concordat is enough in itself? Could you use your power as funders to drive up standards in other ways?
Dr Hill: There are positives to the scope for interpretation, as well as the negatives that your question implies. We have a very diverse higher education sector. Would one suggest that exactly the same processes and requirements should apply to the Royal Northern College of Music and University College London? We need some scope for nuance within that. I agree that, following the UUK review, there are areas where we need to clarify with institutions what our expectations are. We made some endeavours on that when we made it a condition of grant, and there is some guidance that we make available to institutions. However, in areas like publishing narrative statements, making clear who the named contacts are within universities and so on, there is scope for us to be clearer about our requirements.
Q437 Chair: Do you have a plan to set out new requirements in the near future?
Dr Hill: The other thing that is emerging from the UUK review is a new high-level forum to discuss these matters. We intend to discuss them in the context of that forum and then to decide how to move forward.
Dr Peatfield: The research councils fund a lot of research, both in the UK and overseas. We have a responsibility to make sure that our money is well spent. For many years, we have had initiatives to try to tighten up the requirements for that. The terms and conditions of receiving research council grants refer to the fact that researchers have to abide by RCUK guidance and policy, which is fairly detailed on what we expect those who receive our funds to do.
Some of the research councils are also employers of researchers, so some of us have a separate role—just like any university, in a sense—as regards how we manage and train our own staff. As funders, we have quite a lot of power, as we are the source of the money.
Q438 Chair: Do you share Steven’s view that more can be done and that you can use your power as funders more effectively to drive up standards?
Dr Peatfield: Yes. We have done quite a lot in the past two or three years to make some of our guidance clearer. We now have frequently asked questions on our website, to tighten up some of that. There is probably more that can be done, but we have done quite a lot recently.
As was referred to earlier, one thing that we did was ask universities and those we fund to let us know earlier in the process when they receive allegations. It is difficult to ascribe cause and effect, but that appears to have encouraged more reporting to us. The number of cases or allegations has gone up.
Q439 Chair: By how much?
Dr Peatfield: It is difficult to say, because it came in only in April last year, but I would guess that it is has gone up by roughly 50%. The numbers are small, of course.
Q440 Chair: How do research councils and funding councils across the UK work together on research integrity?
Dr Peatfield: We are both involved in the concordat. We have both subscribed to that, obviously. We were involved not only in developing it at the outset, but in the working group that keeps an eye on it and in the review that was done. The research councils have a funding assurance programme, dipsticking universities every year. We share that information. Most of it is to do with financial matters, rather than research integrity, but two or three years ago we added some research integrity questions, which are very specific. It is a dipsticking thing, so it does not give a complete picture, but it is information that we can share with the funding councils. They have a similar assurance process.
Q441 Chair: Do you work to develop common processes and protocols between you?
Dr Peatfield: Not in an active sense.
Q442 Chair: Should you?
Dr Peatfield: We do, in the sense that we are all singing from the same hymn sheet. It is not as if there are any differences between us.
Dr Hill: We work together closely. We currently have separate assurance and audit processes, which Tony has already mentioned. It is worth mentioning that, as well as working very closely with the research councils, historically HEFCE has worked with the other UK funding bodies in Scotland, Wales and Northern Ireland, which have similar policy approaches to us. We have ensured consistency across that. On close working, it is also worth saying that, in a couple of months’ time, Tony and I will be in the same organisation. That gives us some opportunities for increased sharing.
Q443 Chair: Do you think that UKRI provides us with the opportunity for greater co-ordination of processes, protocols and so forth on research integrity?
Dr Hill: We co-ordinate a lot already, but having a single organisation will make that more streamlined and straightforward.
Q444 Chair: But you will maintain the link with the devolved Administrations.
Dr Hill: Clearly, it is a very important role for Research England within the context of UKRI to maintain the links with our colleagues in Scotland, Wales and Northern Ireland. That is definitely very high on our agenda.
Dr Peatfield: It will also bring in Innovate UK, which funds more into the private sector. That may address the question that we had earlier about the extent of private sector interest in this.
Q445 Chair: Sure. RCUK guidelines now require research organisations to report allegations of misconduct to the relevant research council. Will RCUK publish summary data from the information that it receives in that way?
Dr Peatfield: Through the funding assurance programme, we receive information from selected universities. We have published that information on the RCUK website for the past five years. It was one of the requirements in the concordat that we produce a narrative statement, so that information has been published. Obviously, it is a selective sample, so one cannot read too much into the information there. As I said, since April, we have been collecting information from universities more completely.
Q446 Chair: At the earlier stage of investigations.
Dr Peatfield: Yes. Of course, we can ask only for information that involves RCUK funding or RCUK involvement in some form, be it through training, through funding or through referees and panel members, so it will not be a complete UK picture. However, it will be a much more complete picture than we had in the past.
Q447 Chair: Will that be included in what is published?
Dr Peatfield: We will include that information—the high-level numbers, certainly—in what we publish in future narrative statements.
Q448 Martin Whitfield: I want to look again at the concordat. Unlike the first panel, you are both signatories to the concordat. Just to give an overview, will you say what proportion of institutions you think currently comply with it?
Dr Peatfield: There are many aspects of the concordat.
Q449 Martin Whitfield: Hence the use of the word “overview.”
Dr Peatfield: It is difficult to say. As we discussed earlier, there is a very wide range of universities in the UK, which covers some that are research intensive and some that are not. It is almost an impossible question to answer. One would have to be more specific and to drill down into specific questions.
Q450 Martin Whitfield: It was a slightly unfair question. How about we turn it the other way? As funders, have you ever withdrawn funding from an institution that is not complying with it?
Dr Peatfield: The research councils much prefer carrots to sticks. We tend to try to work with the institutions, where there are cases. As we said in our written evidence, often by the time that a case is proven the research grant will have ended, so withdrawing that funding is not feasible. From memory, I do not think that we have prevented an organisation from applying to us for funding because of some previous misconduct arrangement.
Dr Hill: To go back to your first question, we have information on universities in England. I can tell you—this will lead into answering your second question—that in last year’s funding return we had two institutions that were not compliant and declared themselves as not compliant. Like Tony, we favour carrots, rather than sticks. That triggered a process with those institutions that involved very close contact with them, the requirement to develop an action plan and following up that action plan. I am pleased to say that in this year’s return both those institutions are now compliant.
This year we have two different institutions that are not compliant. Both are in the part of the sector that carries out very little research; in fact, one of them does not receive research funding from us. However, we will talk to the other institution and go through the same process, to ensure compliance. The objective here is to develop the culture whereby institutions recognise where they have shortcomings and are open to working with us in order to address them.
Q451 Martin Whitfield: And those institutions and organisations recognise the significance that you place on the concordat and the potential nightmare scenario of not getting funding. We have heard that there are a lot of carrots—a lot of assistance and help—between the two.
Do you think that the wording of the concordat is strong enough as it stands, or would you like to see improvements in it? Would you like to see it develop?
Dr Hill: There are some areas where more clarity would be helpful. We have talked about the publishing of narrative statements, where the wording of the concordat is that universities “should” do that. We are of the mind that we need to make clear that that is not just an option, but a requirement. There are examples like that, but the evidence from the UUK review of the concordat is that it is driving culture change and change in behaviour within institutions. We need to continue to support that.
Dr Peatfield: One thing that is not clear in the concordat is that, in my view, universities should make public on their websites the name of the person who is responsible for research misconduct.
Q452 Chair: It is a recommendation.
Dr Peatfield: Sometimes it is just a job title, rather than a name. I suspect that giving a job title is convenient, because the email address will not change if the individual changes role. However, I think that it rams home the personal responsibility of those people if their names are on the website. That is one area where it could be a bit clearer.
Q453 Chair: And the name for whistleblowers to contact.
Dr Peatfield: Yes. It will be the same person, possibly.
Dr Hill: I very much agree with that.
Q454 Martin Whitfield: Do you think that there should be an obligation on that named person to produce an annual narrative report?
Dr Peatfield: They are supposed to, as part of the concordat, so yes.
Q455 Martin Whitfield: A more useful report, shall we say? A more transparent report, possibly.
Dr Peatfield: In my view, transparency is always good. There are issues about confidentiality and deductive disclosure that universities have to worry about, if they release too much information that appears to be anonymous, but people can work out who is involved.
Q456 Martin Whitfield: RCUK gets organisations to answer six questions about compliance with the concordat. Are those responses published?
Dr Peatfield: Not individually.
Q457 Martin Whitfield: May I ask why not?
Dr Peatfield: It is something that we could consider.
Q458 Chair: You have just said that you support transparency.
Dr Peatfield: I do. It is possible. We have not done it up to now, but we certainly could.
Q459 Martin Whitfield: Do you see that there would be a benefit to publishing that?
Dr Peatfield: The high-level information is published. We publish the numbers.
Q460 Martin Whitfield: I meant the information that drills down, in response to the questions that you put. They are posed for very good reasons.
Dr Peatfield: Yes.
Q461 Chair: Should we not all see whether universities that are receiving public money are meeting the standards for something they have signed up to?
Dr Peatfield: Yes. Sometimes the responses are not black and white, but I agree with you in principle.
Q462 Chair: So that is something that you will consider.
Dr Peatfield: Yes, we will consider that.
Q463 Martin Whitfield: My final question relates to an issue that was mentioned with the first panel. Do you think that there should be an annual report on the state of research integrity? At the minute, you are dipsticking and sampling, rather than doing a fuller report. Do you think that there would be value in that?
Dr Peatfield: Personally, I think that there would be value in it. As I have said before, as funders, we can require only information concerning our own funding.
Martin Whitfield: Absolutely.
Dr Peatfield: In that sense, it would not be complete. However, one could envisage a situation where, if universities complied with the concordat fully, somebody could assemble all those inputs and create an overall report from them. That would certainly be possible.
Dr Hill: I would support that, too, with the same proviso that there is a challenge in how you collect all the information. There is some work that could be done at a national aggregate level, rather than necessarily looking at individual institutions. Maybe the UKRIO study you heard about earlier will look at that.
I go back to the Retraction Watch database. It is assembling a database of retractions, as you have already heard. Retractions are not a perfect proxy for issues of research integrity. They are an important part of the scientific process, and there are many perfectly valid reasons why things get retracted, but research misconduct will certainly feature in them.
I looked yesterday, and the Retraction Watch database now has nearly 17,000 retractions on it. It is possible to mine the data to look at how the UK performs relative to other countries. UK authors appear on papers that account for just under 3% of the retractions. That is set against the UK contribution to journal articles, which is about 6% of the global share. That indicates that the UK is well below what you might expect. It is not dissimilar to the US, where the retraction figure is about 10% and the global share is around 20%. Again, they are doing reasonably well.
You can contrast that with other countries like China. China accounts for 20% of global publications, but nearly 50% of the retractions on the database involve authors from China. Using data like those, you can benchmark national performance, as well as look at individual countries.
Q464 Martin Whitfield: We have spoken a lot about improving the culture of research. Would the elements that we have discussed today, although not answering that, go a long way towards improving the culture of research and, more importantly, research integrity? Do they all form part of the jigsaw of the answer?
Dr Peatfield: Yes, absolutely.
Q465 Chair: Do you share the view that institutions that are carrying out a lot of research and that repeatedly give nil returns for inquiries in their narrative statements should at least give rise to some concern? Would you expect there to be issues an institution should be looking at?
Dr Hill: I go back to the comments by the earlier panel. The current data, until April this year, relates to formal investigations, rather than the screening phase. If I can turn your question around, I would be more concerned if that broader definition of what needed to be reported led to zero. If nothing was even being screened for investigation, that would be concerning, but I am not sure that seeing no cases that go to formal investigation is necessarily concerning.
Q466 Chair: That begs the question, should the institution’s annual report, which the concordat encourages, include those early-stage investigations, so that there is a full picture of what is happening? In a lot of institutions, we are just seeing nil return, nil return, nil return. That gives rise to some concerns.
Dr Hill: Certainly, the RCUK changes are moving in that direction. I am supportive of that.
Q467 Chair: Of the narrative statements each year showing the full picture.
Dr Hill: Yes.
Q468 Carol Monaghan: We have been told in one submission that pressures of the REF contribute to problems with the research culture. Do you agree?
Dr Hill: This is probably a question for me. I agree that there is a perception and that there are potentially conditions within universities that relate the REF to pressure to publish. However, we have to look critically at the features of the REF and the extent to which they are really linked to pressure to publish.
The first comment that I would make is that the REF is a UK-specific activity—it does not apply in other countries—yet we see that the UK performs quite well, both in terms of the retraction data I have just talked about and in terms of the quality of publications. A context where there is strong pressure to publish tends to lead to an increase in the number of publications, with a commensurate decrease in their quality. There is some evidence for that in other countries. However, what we have seen in the UK, from the ways in which we can measure that, is an increase in quality over time. That suggests that the UK is not in a “pile ’em high, sell ’em cheap” mentality when it comes to research publications.
In the REF itself, there is a strong focus on things other than publications. In the last REF, 20% of the outcome was focused on broader societal impact and 15% was focused on the research environment—which, by the way, includes considerations of research integrity. In the previous exercise, the publications element—what we refer to as the research outputs element—required researchers to produce only four publications over a six-year period. In many disciplines, that is quite a low requirement, in terms of the total volume. In disciplines where long-form research outputs are more common, we have provisions that allow double-weighting of outputs, which reduces the number further. For the future, we are taking the average number per individual down lower, to 2.5 research outputs per individual, on average.
Q469 Carol Monaghan: Is that annually?
Dr Hill: No, it is over the whole period. That is a seven-year period, in the case of REF 2021. The fact that we are talking about an average per person reflects something that Sir Bernard mentioned earlier—that we are moving away from the REF being very strongly linked to individuals, and their research outputs, to looking much more consistently at the output of research units and groups.
Q470 Carol Monaghan: So you are pushing more for quality than for quantity with those numbers.
Dr Hill: Yes. To be frank, the REF, and the RAE before it, has always had a strong pressure towards quality, rather than quantity.
Q471 Carol Monaghan: May I push you a little more? You said that an institution’s research integrity was included in the judgments. In what way is that done? Is there scope for making research integrity a larger part of the REF judgments?
Dr Hill: Yes. In the 2014 exercise, two of the broad subject groupings took into account matters related to research governance and integrity in the environment section. Those were the grouping covering medical and life sciences research and the grouping covering social science research.
Q472 Carol Monaghan: Who judges that?
Dr Hill: It is judged by the panel members, who are members of the academic community, together with users of research. In the medical sphere, for example, you would have practitioners, patient groups and others represented.
For the upcoming exercise in 2021, we are currently at the stage where we are assembling the panels and they are coming up with their criteria. We will require all of them to consider how research integrity and misconduct issues can be covered in the research environment section. That will be a new requirement. It will need to be looked at in the context of the different disciplinary communities. Clearly, what matters in the space for medical research is rather different from what matters in research in the humanities. However, we will require all the panels to consider that and to assess it as part of the exercise.
Q473 Carol Monaghan: I am slightly concerned by something you said. Perhaps I am being overly-concerned, but you said that the judgments had been made by other academics and those using it. Is there not a potential conflict of interest there? Surely an independent judge would be better.
Dr Hill: Those panels are made up of respected people from their fields. The judgments require that expertise. We have very strong processes in place to avoid conflicts of interest where panel members’ own universities are being assessed—they would always leave the room and not be involved in those discussions. These processes are not unlike those used by the research councils. When the panels assemble, they are making assessments on behalf of their disciplinary communities and making judgments between institutions, so they have a strong incentive to do that in a robust way that makes clear where there are strengths and weaknesses.
Q474 Carol Monaghan: You mentioned that part of the REF was looking at the wider societal impact. One submission we have had refers to “… disproportionately rewarding high-profile outputs that are the kind of flashy, breakthrough-type research.” That was a comment about the REF. Does the REF make it difficult to secure funding for replication studies and meta-research—research into research methods?
Dr Hill: I do not think it does. The panels judge research outputs against three criteria: originality; significance; and rigour. Those criteria are applied to all sorts of different types of outputs, not just journal articles: meta-reviews and different pieces of work.
Q475 Carol Monaghan: I think the comment was about how you prove the societal impact if all you are doing is researching research.
Dr Hill: I think we have two elements of the REF. We look at the research outputs that are judged against their academic impact and the difference they are making in their disciplines in advancing knowledge, and we have a separate section that looks at the broader societal impacts. The section that looks at the broader societal impacts looks back over a 20‑year horizon rather than focusing on just the assessment period, and it is looking at all different types of societal impacts.
There is a huge diversity within those impacts submitted for assessment. Drug discoveries will potentially come from research outputs in the past that were very high profile and regarded as being huge in their discipline but have led to drug development. You also have examples that are on a different scale and involve research that can be very local and focused on particular problems and communities, so there is a huge diversity.
Q476 Carol Monaghan: I understand that, but the problem is that, if you are not able to show, even on a local scale with a small project, that this research has had an impact, there will be difficulties. That brings into question who is going to fund the research into research methods, which is what we are talking about today.
Dr Hill: Research into research methods, if it meets the criteria of originality, significance and rigour, could be submitted as part of the REF exercise and would score very well, so there is no barrier to that sort of work.
Q477 Stephen Metcalfe: You will have heard Professor Silverman say earlier that he was generally supportive of the idea of some formal qualification in research. Would you consider introducing some form of licence or qualification to demonstrate that you understand what research integrity is, which could be withdrawn if there was a problem? It is a bit like the GMC striking a doctor off the register.
Dr Peatfield: It would be very bureaucratic and difficult to implement. Research is a worldwide activity. If you are recruiting somebody from Germany to the UK, what would you expect them to bring in the form of licensed activity?
At one level, a PhD is deemed to be a scientific qualification that defines you as a scientist. That harks back to what we discussed earlier about PhD training and the extent to which research integrity is included within it. It is possible to set up the equivalent of the General Medical Council with a register of researchers. There would be huge turnover with people coming and going. You would have to have criteria in addition to those people must have to get their PhDs. Therefore, those criteria might be very different for a clinical triallist compared with an astrophysicist. How do you judge those criteria? Who decides?
You have to have a process for striking off people. Somebody would have to complain, and then you go through a legal process to strike them off, because you are depriving somebody of the right to work. My personal view is that it would be extremely bureaucratic and expensive to set up and probably will not work very well.
Dr Hill: I tend to agree with the analogy with the GMC. There is a clearly defined body of knowledge and practice related to being a medical practitioner. It is rather difficult to define what that would be across the spectrum of research. As Tony said, even within the sciences there is a huge amount of diversity, and when you move into the social sciences and humanities that diversity is greater.
I agree with Sir Bernard that more training for PhD students and researchers across the spectrum in matters of research integrity would be a good thing. We already have another concordat on research and development that universities have been implementing for a decade. Associated with that is a Researcher Development Framework that does indeed include those issues, so we already have some soft measures in place that encourage that training. It is also part of the research integrity concordat. I think it is an area we should look at in firming up the potential.
Q478 Stephen Metcalfe: I hear what you say about a licence to research, which we might want to explore further. Will you explain what sanctions are available to you where you find there has been a problem with research?
Dr Peatfield: A lot of the sanctions are local, so the employer does the investigation and is the body that decides whether there has been a proven case. Therefore, the primary sanctions are from the employer. Certainly, within the research councils as employers of researchers it is part of the contract. If somebody is proven to have committed misconduct it is a sackable offence, and I think it is the same with universities. Therefore, the first line of sanctions is with the local employer.
Funders can apply sanctions, and I have listed in our written evidence what the range is. We can terminate a grant if it is still running; if we wanted to, we could stop a clinical organisation applying for new grant, but we have never done that.
Q479 Chair: Do you ever do it?
Dr Peatfield: I do not think we have ever stopped an organisation applying for a grant. On occasion we have gone back where there has been a case against an individual and worked with universities to take that individual off the research project, or ensure much better supervision within the research project. We have acted more with a carrot than a stick to try to make sure the research continues, because we have funded it up to a certain point and it is in our interests and the researchers’ interests to try to draw it to a satisfactory conclusion. Therefore, we work with universities to try to do that.
The one case where we applied a sanction arose because somebody was found to have plagiarised. They were a Research Councils referee and we removed them from the list of potential referees for that reason.
I am not a lawyer, but I understand that it may be illegal to blacklist people if it stops them working, so blacklists per se are not an option.
Q480 Stephen Metcalfe: There is no central list of those who have had funding withdrawn.
Dr Hill: No.
Q481 Stephen Metcalfe: On the basis that you assume it may be illegal.
Dr Peatfield: We have never actually done it within Research Councils, but, looking to the future, if we did we would have a record of the fact we had done it.
Q482 Stephen Metcalfe: You are not keeping a blacklist. You have the ability to withdraw funding. I think you gave us one case. In 2011 the previous Committee was told that there were no cases where funding had been withdrawn on the grounds of fraud or misconduct in research. Has that changed somewhat?
Dr Peatfield: It has not changed. We have not withdrawn funding; we have worked with institutions to ensure that the funding we have provided has been used properly.
Q483 Stephen Metcalfe: Where problems have arisen with the integrity of research are they being reported quickly enough for you to be able to withdraw funding?
Dr Peatfield: That was one of the reasons behind the requirement that universities inform us as soon as they have decided there is a potential for research misconduct, as opposed to some other kind of misconduct, because quite a lot of these things arise from a breakdown in personal relationships. Therefore, once the university has decided that there is a possibility that there is research misconduct that is the stage at which we would like to be informed, so we should be much better able to track the time it takes from that first notification to an outcome. Despite the fact that everyone wants to do something quickly, for all the right reasons, setting up the investigation panels, making them meet and getting all the evidence together takes time.
Q484 Stephen Metcalfe: Do you think any of this will change with the creation of UKRI?
Dr Peatfield: On day one, no. UKRI is concentrating on a lot of other things that need to be in place for day one, but I know it will be taking it seriously—Mark Walport himself has said so. Obviously it will want to look at bringing in Innovate UK and Research England under the same umbrella. I cannot speak for it in any detail, but I am sure it will take it very seriously and try to improve things.
Q485 Chair: There is a problem, is there not, where somebody departs from a university with a compromise agreement and non‑disclosure clause that keeps everybody quiet about what has happened? That researcher then crops up somewhere else and again applies for public money to do research. How do we address that problem? Should it be acceptable for universities to allow someone to leave with a deal that maintains confidentiality and effectively covers up misconduct in research?
Dr Peatfield: Probably not, but it may be quite difficult to put into practice, in that if there are subsequent retractions they will make the outcome public. Even if a university may think it has reached some sort of compromise agreement and there is a subsequent retraction, it will become evident that that particular researcher has been committing misconduct.
It crops up occasionally where somebody just moves from one institution to another. A report by Science Europe last year recommended that universities employing researchers should ask the question at interview, “Have any cases of misconduct been held against you?” If that person lies, that would be a reason for subsequent dismissal if they were then appointed. There is an onus on the new employer, or any employing institution, to ask those who are applying for jobs what their history has been.
Q486 Chair: Perhaps that is one way the concordat needs to be tightened up.
Dr Peatfield: Yes.
Dr Hill: I add two comments to that. First, I agree it is an issue. It is also an issue for poor practice and behaviour in many other parts of society.
Q487 Chair: It does not mean it is not important.
Dr Hill: Indeed, but I do think we have to keep that in context. Secondly, on retractions one of the weaknesses in the system is how poor we are—when I say “we,” I mean primarily journal publishers—in collating information around retractions. It can be quite hard to find out whether a piece of work has been retracted and who has had papers retracted. The work that Retraction Watch is doing is great, but I still think it is not always 100% transparent. When you do a search and find a journal article, it is not always 100% transparent that there has been a retraction or modification.
Dr Peatfield: Or that the retraction has been due to some kind of misconduct, or even sloppy science. Papers are retracted for quite good reasons—for example, that the wrong regimen was used on the day and there was a mistake in how the research was done, which may be purely accidental. Therefore, one has to be careful about how one connects research misconduct with retractions.
Q488 Vicky Ford: We keep focusing on universities as research centres, but a lot of research in areas such as genomics is done in the NHS. The relationship between doctors and pharmaceutical companies is an area where one may be thinking about training doctors in medical research. We have talked about other organisations, not just universities. Is that an area?
Dr Peatfield: Being a medical practitioner, there are a lot of ethical and other issues related to research integrity and misconduct as part of that. Doctors are in a different position for all sorts of reasons. I would refer you to the ABPI, which recently tried to create a register of doctors’ interests and their relationships with pharmaceutical companies. I think that attempt at transparency has been helpful.
Q489 Bill Grant: Training has been touched on. A key player in research, as will be well known to you, is the Association of Medical Research Charities. I sense they may be funders and very important players. It has advocated a role for centralised training on research integrity, as opposed to what would appear to be a fragmented system with no centralised training. Do you agree that there is a role for centralised training around integrity? If so, who would be best placed to deliver such centralised training?
Dr Hill: There is probably a core of activity where consistent training would be helpful. It would not necessarily be the best outcome to have a single provider, but potentially UKRIO could develop some guidelines around what that core curriculum might be and other organisations could provide that training. That would be my view.
Q490 Chair: They are listening to you.
Dr Peatfield: I would agree. The other thing is pitching the training at the right level. If you have a first-year PhD student, it is quite a challenge to pitch the training at a level that will engage them, versus somebody who is a four-year postdoc or a reader at a university. How you pitch it and relate it to their own actual research in which they are interested can be quite a challenge. One could keep it at a high level and talk about fabrication and falsification—one can write down “read”—but how they relate to their own research practice I think has to be a bit more bespoke.
Q491 Bill Grant: Should training be made compulsory? “Compulsory” is not a particularly nice word. Do you see people moving away from it being compulsory?
Dr Peatfield: It is compulsory in the sense that part of the concordat referred to is that universities or recipients of our funding are required to provide training, and we specify quite tightly what that training has to cover.
Q492 Chair: That includes integrity.
Dr Peatfield: Yes, research integrity. It is well specified what is expected at PhD level. There is a separate question about how we monitor that. There is a dipstick monitoring process, but obviously we cannot have comprehensive policing of what actually goes on. One of the things we have done is to try to focus our doctoral training in fewer places, and that makes for better training but also makes it easier to keep an eye on what is going on.
Q493 Bill Grant: We touched on centralised training. You would not advocate a single provider, but, on balance, as between some form of centralised training and local community-based, or university-based, training, what would you say is the best way forward to improve the integrity of research?
Dr Hill: I would agree with Tony’s point that there is a common core of material and issues, but it needs to be contextualised for different disciplines and areas. Because of that, on your spectrum of fully bespoke and centralised training I would be closer to the bespoke end of it.
Q494 Bill Grant: One size does not fit all when it comes to research integrity.
Dr Hill: Yes.
Q495 Chair: You have indicated that you are considering more steps to use your influence and powers as funders to tighten rules on integrity. We recognise that it is a serious problem. It may not be a widespread problem, but when it happens it is serious and it has an impact on people’s trust in research and so on. In guiding us in the drafting of our report, are there any other recommendations you would put forward for what should be done to improve standards of research integrity in this country?
Dr Hill: I would echo a comment made by Sir Bernard about increasing the openness of research data and practices. I would say that policies further to enhance that openness of research processes and practices have a really important role to play.
Q496 Chair: And indeed to require openness.
Dr Hill: Indeed. We have yet another concordat on open research data which has as a basic principle the presumption that it should be made openly available, and then there are some reasons why you might not do that: confidentiality, national security and so on. Therefore, we are already moving towards a framework where we expect research data to be made open. We will be looking at that in the next REF; we will have an element on that within the research environment. There is evidence, which we cited in our written material to you, that there is at least a correlation between researchers who are prepared to share their data and high-quality statistical practices, for example. That is a really important area to emphasise.
Dr Peatfield: The quick answer to your question about what other things you might recommend is: not really. To follow up Steven’s point, a lot of it is to do with culture, and we try to move things along in various ways using the tools we have, but I have a bit more than a feeling that things are moving in the right direction. For example, with open data, as Sir Bernard said, 30 years ago people would have said, “Why would I want to share my data? It’s mine.” I think that the younger generation see the benefits of sharing data in what they can get back from other people’s research. Therefore, a cultural shift is happening and the more we can accelerate that the better. Things are definitely moving in the right direction in openness and transparency.
Chair: May I thank both of you very much indeed for your time? It is enormously appreciated.
Examination of witness
Witness: Professor Montgomery.
Q497 Chair: Will you introduce yourself?
Professor Montgomery: I am Jonathan Montgomery, non‑executive chair of the Health Research Authority. In the light of the evidence you are taking, I should also say that my day job is at University College London. I know you have received evidence from them, but I have not been involved in it. Until last February I chaired the Nuffield Council on Bioethics. I know you have received evidence on its research culture work of which I was part.
Q498 Chair: The Committee wrote to you following its session in December on clinical trials transparency. In the letter I asked whether there were any barriers to the HRA publishing all of the information it held on which clinical trials had reported and which had not. It seems to me that that is a central issue of importance. Your response does not actually address that point, which obviously disappointed me. Could you respond now?
Professor Montgomery: I apologise for that, Chair. We hold data that come through in relation to projects, some of which is commercially sensitive, and within our processing systems we need to respect the basis on which people have given that to us. That was the reason Dr Kolstoe’s work did not name people.
However, going forward in relation to whether trials have resulted in publication, I see no reason why we cannot match publicly available data. We report on all the trials we have approved and try to match them—if I may, I will come back to the challenges in doing that—with publicly available data. We would have no difficulty in putting the two things next to each other so people can share the comparisons.
Q499 Chair: Surely, these confidentiality requirements that you say constrain you in publishing outcomes conflict with the position, which I think has been in place since 2014, that there needs to be reporting of the outcomes of research. Should this not be in the public domain, and why are you not requiring that it should be?
Professor Montgomery: If I may break down the question, we do not hold the outcomes of research. We hold the applications that come to us to get permission. We publish a summary of the applications that we approve. We have an end-of-trial report that has to come in to us in which we ask people to identify publications and headline-level findings, but they do not hold the results of the reports in the way that, say, a clinical trial report would be sent to the MRHA, the regulator.
Therefore, we have a gap in the information we hold that we need to seek to plug. We may be able to get ourselves to that position. We are in the process of starting to re‑provide our application system. We hope that will enable us to create a field in the process where some refindings could be put. If we were able to do that, we would be able to publish, alongside the research summaries of approved studies, the summary of findings as well, but we do not hold that information at present.
Q500 Chair: How would you respond if this Committee requested or required you to disclose what you have already received?
Professor Montgomery: I would need to check whether there were any particular legal barriers at the level you would be asking for. I do not think there would be any issues of patient confidentiality, although there would be about patient level access. In relation to the final study reports, as far as I am aware there may be questions of commercial sensitivity that would require redaction, but I am not sure of the details.
Q501 Chair: May we put to you a request to disclose everything that is included in this?
Professor Montgomery: Do you want to see the actual reports from each of the studies or a summary? I do not know the details, so I am not sure how tricky or voluminous it is.
Q502 Chair: We would like you to go away and look at it and see what you are able to disclose to us, because it is an important matter.
Professor Montgomery: I can certainly take it away and see what we have in the final reports that are not in a standard format but are asked to include main findings.
Q503 Chair: Should they be in a standard format?
Professor Montgomery: I think they will need to be as we re-provide our new data system. I would hope that will make things more easily comparable, so I agree they should be.
Q504 Chair: Would you agree with Dr Goldacre that universities with a poor track record in complying with the requirement to publish trial results should be named and shamed, given that it is a clear requirement?
Professor Montgomery: There are challenges in doing that. I am not in favour of naming and shaming; I am in favour of creating a more productive environment. We would be very keen to make transparent who has and has not published, but we would like that to be a prompt to publish, so we would want to be working with the universities to encourage publication.
Q505 Chair: There is a danger that all of this sounds a bit cosy. Since 2014 there has been a requirement to publish the results of research, and yet, according to Dr Goldacre’s evidence to us, compliance rates in academia are extremely poor and variable; some universities are up to 75% reporting; many UK universities are down to 10% or 0%. Is it satisfactory for you just to say you are prepared to work with universities? Should they not be complying with this requirement?
Professor Montgomery: I think they should. Perhaps I could share a little bit of the reason we have not progressed to that stage as we thought we would. We have to identify what counts as reporting and publishing. If you look at the evidence you received from the Medical Research Council, it had a very high publication rate four years after studies closed, but it found that only half of the publications actually reported the main findings as were predicted.
Our ability to know what is reported is quite tricky. We need to understand the “shaming” bit of the transparency process. Would we want to shame people who have published important sub-results but not the main results? I think we would want to make those things apparent. Our chief executive, Teresa Allen, who is behind me, has met Ben Goldacre and the team. The software he has been developing, which enables us to make transparent the range of outputs available publicly through the internet and match it to the studies that have been approved, gives us a more nuanced sense of what has come out.
Therefore, I am nervous about our ability to audit against a standard we do not fully understand to see whether people have complied, because to do that is quite an intensive process. I think it would be relatively easy to audit complete non‑publication, but our experience of the audit and registration is that we have managed to increase the rate of registrations by going back to people and saying, “You don’t appear to have registered,” and that has prompted them to do the right thing.
Q506 Chair: Do you agree that, four years on from a requirement coming in, such low rates among some universities is unacceptable? What are you doing to change that so we have 100% compliance? Surely, institutions should not be receiving public money. It is important to look at the purpose of this. Non-publication of research distorts public knowledge, does it not, and potentially undermines public trust in research? This is public money that has been invested in research. Do we not have to move rapidly to a situation where everything is published, as is required?
Professor Montgomery: I would agree with that. I also think that non‑publication betrays the patients and participants in the studies.
Q507 Chair: Yes. So what are you doing to make sure it happens?
Professor Montgomery: We are finding this much more complicated than we imagined. I have indicated the challenge of what counts as a publication. We have commissioned some work from the EQUATOR Network—I think you had some evidence from it—which proposed a redesign of the information we ask for in the applications that will grade more carefully the various modes in which it is planned to report results. That will work its way into the re-provision of the application system and give us a benchmark against which we can follow up whether people have done it.
We are seeking to incorporate good standards in reporting. You will be aware that there is a series of protocols that guide people on the appropriate designs of their original protocols and their reporting. We are proposing to build them into the question that is asked of the applicants, which will then give us something to follow through.
The biggest challenge I have not spoken about is working out whose performance we should be monitoring. We get applications from investigators. You have spoken about the problems with universities that are, in our terminology, typically, but not exclusively, sponsors. In relation to the misconduct as opposed to the poor practice end, our research policy framework visits the responsibilities for dealing with that on employers, so we need to work out how we explain the failure to publish in relation to each of those groups.
My understanding is that Dr Goldacre’s software will enable us to cull those things separately and will lead to the ability to report, but that would lose the link with study-by-study approval, which is the basic currency in which our IT systems work—so this turns out to be more complicated than we imagined.
We have a transparency forum that brings together publishers, researchers, higher education institutions and industry. That is working on how we should move that forward, but we have not made the progress I would have anticipated.
Q508 Chair: If HRA’s publication information is incomplete could it demand that the information is supplied by the institution?
Professor Montgomery: In relation to the follow-up of our registration audit, which is more fully developed than the work on publication, that is exactly what we do. If we cannot identify the registration, we go back to the applicants and challenge them for non‑registration, and in most cases that results in an account of why registration was not needed—most commonly because trials have not actually started—or they take steps to register.
Q509 Chair: But you could demand anything that was missing from the results of the research.
Professor Montgomery: We could demand the final study report.
Q510 Chair: Are you going to?
Professor Montgomery: We already ask for that. I will take away the question of what is contained in that. I am not sufficiently familiar with the level of detail we are given.
Q511 Chair: Could research ethics committees make it a requirement of further approvals that the researcher or funder has a complete publication record of previous results?
Professor Montgomery: I think that would be tricky. We did some work in 2013 on exploring with our research ethics committees their willingness to take forward Dr Goldacre’s proposal at that stage, which was similar but rather more abrupt in time to the discussions we have been having. We took to our research ethics committee members the question whether we should make it a condition of a new application that results had been published within 12 months of completion of the previous ones. There was quite strong resistance to that.
Q512 Chair: Why?
Professor Montgomery: There was a group that simply told us it did not think it should happen; a group that thought it would not be possible to deliver on it; and a group that thought it was too ambiguous. Broadly, those are the pieces of work I described that we are doing to understand better what to audit. Can we clarify what counts as a publication?
Q513 Chair: Other than resistance, there appears to be no good reason for it.
Professor Montgomery: Let me separate the Health Research Authority role and the ethics committee role. I was not present at the meetings; I have seen only the reports. Some resistance has been expressed to the ethics committee judging one application on the basis of others. We had this when we introduced the requirement of trial registration for phase 1 trials. There was an argument that it was unethical to turn down an otherwise well-designed study.
Q514 Chair: Therefore, someone can carry on not publishing.
Professor Montgomery: That is the difference between what the Health Research Authority should do and what the ethics committees should do. The Health Research Authority is already responsible for assuring compliance with a number of regulatory and legal requirements. Once we have got to the stage of clarifying what the expectation is I anticipate that Health Research Authority officers will take responsibility for following up that compliance and making it available to ethics committees as part of its consideration, in the same way as we would in relation to compliance with radiation standards or the like.
I see the key decision makers as not so much the ethics committee as responding; it is the hospitals that are prepared to work with researchers and sponsors taking responsibility for using that information in the projects they are prepared to fund. I am not convinced that the ethics committee is the right bit of the system to do that. I am thoroughly persuaded that we need to get the system to be able to make that sort of judgment.
Q515 Chair: What scope is there for the HRA imposing sanctions when the results of trials are not published? Does it have sufficient teeth under existing legislation, or does it need more powers?
Professor Montgomery: We are essentially the gatekeeper to admission to research, which is a privilege, and people need to demonstrate they are appropriate to do it. We already have requirements to consider the CVs of researchers and ensure that staff are appropriately trained for, say, good clinical practice if they are doing interventional medicinal products.
I think we would have the ability to draw attention to what should be on a CV. We do not have sanctions as part of the HRA, other than refusing permissions, but we would refer to regulatory bodies. We have referred people to health professional regulators where we identify breaches. Once we have a clear standard, I think we would be able to record breaches. The board receives a regular report on breaches of conditions, and to that extent I think we could become part of the transparency of whether people had complied with their conditions. We do not employ the researchers, so our only real powers are to withhold permissions or refer to those who do have those powers.
Q516 Chair: That is potentially quite a considerable power, if you use it.
Professor Montgomery: It is a power that is visited on individual researchers. It may well be that they are not the key part of the problem. We would look at good practice and the work of NIHR, which has created its own location where all its studies can be published, even if they are not going into peer review journals, and the work of the MRC. Industry’s reaction to the registration requirements has been important. I would agree with Dr Goldacre that academia still has quite a long way to go.
Q517 Vicky Ford: In your letter you suggested that this could cost about £2.4 million, but reading between the lines it sounds as if you might have been doing quite a lot of work with Dr Goldacre to see if you can do it in a different way. How confident are you about that figure?
Professor Montgomery: I do not think we would ever secure that figure. I do not think it would be a sufficient priority against all the other things we are doing if we did have that money to deal with it. I am currently advised, based on early conversations, that we should be able to use an IT-type base and, with four or five staff, we are probably talking about £250,000 to move this forward.
Q518 Vicky Ford: If we were writing recommendations, it might be helpful for us to investigate an IT-based solution first.
Professor Montgomery: I think it would be extremely helpful for you to reiterate why transparency is important; the reasons we need to be able to monitor whether people have reported their findings; and that you are pleased to hear that there may be an affordable IT-based solutions.
Q519 Vicky Ford: Is there a risk that, if the results are not compared with the original intentions, funders will give only partial results and will try to tick the box that they have reported it, but only half?
Professor Montgomery: There is definitely a risk, but that is already apparent in the current system. I hope that the developments we are talking about, which are more nascent than I would like them to be, will get us to a situation where we can separate out the report on findings, which could be in our system based on the revised questions we ask, so you could match back and go alongside the broader question of whether the public has been given access to the information. In our preliminary work, peer review publication is only one of the mechanisms people use. Sometimes it is more appropriate than others. We have significant evidence of conference proceedings that make their way into journals but are not as peer-reviewed publications.
I think we need to keep separate the basic obligation to produce a final report that matches the permission that was given. I think we should be able to hold that within our new systems. That will not remove the need to do the sort of work we are talking about with Dr Goldacre’s team, or the work Dr Kolstoe did, which needs to track whether the full response to the research has been seen.
Q520 Vicky Ford: I recall the suggestion of a sanction. If it was found that researchers had not been putting in place final reports, they may not get future grants, or it might affect their ability to get future permissions. Would you agree with that as something worth considering?
Professor Montgomery: I would expect sponsors to be interested in whether their money had been well spent previously and led to the reported outcomes. I would expect the hosts of research to want to know it was worth hosting that research, so we should be providing them with the opportunity to ask whether they trust a particular research group or sponsor to deal with it. I am not yet quite clear how we would do this, but I would also want to make it possible for potential participants to factor into their decision whether they want to be part of a trial and whether they have confidence that doing it will be worth while because people see it through to fruition. We know that they already trust some organisations more than others. We could provide some information that would help inform those decisions, but we are a little way off being able to do that at the moment.
Q521 Vicky Ford: The registration of clinical trials has been a requirement for ethical approvals for many years. Are there still unregistered phase 1 clinical trials?
Professor Montgomery: In relation to phase 1 clinical trials, our last audit found four to which we could not get a response and information about. All the others were registered. Not all were registered in the databases on which we initially did our internal research, so for some we learned about registration only when we went back to the ones we did not know about. There were four at the time of the audit to which we did not have a response. I am not sure what the current state of play is, but I will find that out.
Q522 Vicky Ford: Is that four out of thousands?
Professor Montgomery: No.
Q523 Vicky Ford: Hundreds.
Professor Montgomery: The total audit looked at 617. From memory, we are talking about 70-odd phase 1s. I could check that. We can provide you with a copy of that audit report.
Q524 Vicky Ford: Are you saying that going forward those four would not be able to slip through the net?
Professor Montgomery: We do not know whether they slipped through the net, so I need to find out whether we have now had a response. I was looking at the audit report that we received at the board.
Q525 Vicky Ford: So you are investigating it.
Professor Montgomery: Yes.
Q526 Vicky Ford: And there should be none.
Professor Montgomery: There should be none. As far as we are aware, all the clinical trials of investigational medicinal products are now registered. We have not found any information on phase 1s to show they are not registered, but we have these four on which we do not have information.
Q527 Carol Monaghan: May I go off on a slight tangent and talk about transparency? You have talked about the publication of results, but I am slightly concerned about transparency in terms of the authors. To give you an example, major questions are now being asked about the efficacy and processes under which the PACE trial for people suffering from ME took place. One of the authors of the trial has a double role as an adviser to the DWP. Is there any move at all to look at those actually conducting the trial and writing reports to see whether they have other interests or whether there are conflicts?
Professor Montgomery: I am not quite sure what stake the HRA has in that, because that happens mostly after our purchase on the submission—
Q528 Carol Monaghan: I am talking about possibly looking into the authors.
Professor Montgomery: As you heard earlier, different disciplines have different authorship conventions. In relation to medical journals, there is a standard protocol on how authorship is expected to be done by agreement among medical journal editors. I am aware separately that, out of the work done by the Academy of Medical Sciences on how we should all use evidence, which was published last year—I was on its steering group on behalf of the HRA—a working group has been convened by Sense about Science, of which I am part, looking at declarations of conflicts of interest. That is not particularly in relation to authorship; it is more generally in relation to the conduct of trials, but I can certainly take back that sort of question.
Q529 Carol Monaghan: That is one example, but there could be others where people conducting trials have an economic interest or otherwise in the results.
Professor Montgomery: That can always be the case. I cannot recall—I will check—where in the IRAS application form we ask about conflicts of interest. I will get confirmation on that.
Q530 Carol Monaghan: It would be useful for people to see them published as well.
Professor Montgomery: The question of authorship of reports is trickier, because a wide range of people may be involved in writing up the report beyond those who got permission to undertake the study. We will be notified of key investigators, but, if they are getting writing support and statistical advice, the modern conventions on authorship seek to give credit for a very wide range of contributions to the writing up. It is an extremely complicated question. What I can take away is: what are our processes for getting declarations and conflicts of interest in relation to the application process? I will certainly do that.
Q531 Chair: Going back to the steps you are taking in discussion with Dr Goldacre and Dr Kolstoe, given it has been four years since the requirements about publication of results of research came in, what is your timescale for getting this resolved and reaching a point where you would feel confident that there was 100% compliance with that requirement?
Professor Montgomery: I do not think I could commit to 100% compliance without bottoming out a number of these complexities. Our timescale for being able to link the final study reports and make them available will depend on the procurement of our new IT system. My understanding is that we got approval of the business case for that within the past few weeks. If I may, I will take that away and write to you with a proposal.
Chair: If you could come back to us, it would be very helpful. Thank you very much indeed. We appreciate your time.