Science and Technology Committee 

Oral evidence: Research Integrity, HC 350

Tuesday 24 October 2017

Ordered by the House of Commons to be published on 24 October 2017.

Watch the meeting 

Members present: Norman Lamb (Chair); Vicky Ford; Bill Grant; Darren Jones; Stephen Metcalfe; Stephanie Peacock; Graham Stringer; and Martin Whitfield.

Questions 1 - 114

Witnesses

I: Professor Dorothy Bishop, Professor of Developmental Neuropsychology, University of Oxford; Dr Arnaud Vaganay, Director, Meta-Lab; and Professor Dame Ottoline Leyser FRS, former Chair, Steering Group on the Culture of Scientific Research, Nuffield Council on Bioethics.

II: Dr Elizabeth Wager, Publications Consultant, Sideview, and Honorary Co-ordinator, the REWARD Alliance; Professor Sir Ian Diamond, Research Policy Network, Universities UK; Professor Ian Walmsley, Pro Vice Chancellor for Research Working Group, Russell Group; and Dr Peter Wilmshurst, Consultant Cardiologist, Royal Stoke University Hospital.

Written evidence from witnesses:

– Professor Dorothy Bishop

– Meta-Lab

– Nuffield Council on Bioethics

– Dr Elizabeth Wager

– Universities UK

– Russell Group

– Dr Peter Wilmshurst

Examination of witnesses

Witnesses: Professor Bishop, Dr Vaganay and Professor Dame Ottoline Leyser.

Q1                Chair: Thank you very much indeed for coming. Would you start by introducing yourselves, just giving your name and your organisation?

Professor Bishop: I am Dorothy Bishop. I am based at the University of Oxford, where I do research on children’s language disorders funded by the Wellcome Trust and the European Research Council.

Professor Dame Ottoline Leyser: My name is Ottoline Leyser. I am a plant developmental geneticist from the University of Cambridge, but I am here today representing the Nuffield Council on Bioethics, for whom I chaired a project on research culture a couple of years ago.

Dr Vaganay: My name is Arnaud Vaganay. Unlike my two colleagues, I am a social scientist. I am a meta-researcher; I do research on research. Today I represent an organisation called Meta-Lab, which does research on research.

Q2                Chair: We will ask some questions to individuals and others to the whole panel. Don’t feel that you all have to respond to everything, if you don’t want to. I will start by asking how you would define research integrity. Does it mean different things to different people in different sectors?

Professor Bishop: It does mean different things to different people, but the key distinction is integrity of research, which to me would refer to the extent to which research is solid, buildable on and properly done, as opposed to integrity of researchers. It is very important not to conflate the two, because research that lacks integrity does not necessarily mean that the researchers lack integrity.

Q3                Chair: They have done their best, but there was just a flaw.

Professor Bishop: They might not have quite done their best—we may come back to that—but they have done stuff without ill intent. Then you have cases where the researchers themselves lack integrity and either plagiarise, or in the worst case make up data and things like that, which is obviously a much more serious issue.

The difficulty is that researchers get very twitchy. If you start talking about research integrity, it is sometimes taken to mean, if there are problems—

Q4                Chair: That there is necessarily wrongdoing.

Professor Bishop: Yes, and that is terrible; it stops people doing anything about it, because they do not want to talk about it.

Dr Vaganay: I still think it is useful perhaps to start with the definition. In my understanding, research integrity is the extent to which researchers comply with the norms of science. In my opinion, it is a synonym for professionalism. What does it mean to be a professional researcher? There are two dimensions to research integrity. There are the technical aspects. Do researchers make specific decisions in the way the profession would expect them to? For example, do they sample a group of people who are representative of the population, and how do they make that decision?

There are also ethical and moral considerations. For example, what are the values upheld by individual researchers and are those values in line with the values of the scientific community? To give you an example, scepticism is a core fundamental value of science. Researchers have to expose their research to scrutiny. Likewise, as researchers we have a duty to scrutinise our colleagues’ research. That is an example of ethical and moral values. There is a technical dimension and a moral dimension.

Q5                Chair: Are you really talking about the behaviour of researchers?

Dr Vaganay: That is correct.

Q6                Chair: There is a separate issue that you are talking about, Professor Bishop, which is the quality of the research that emerges from it.

Professor Bishop: Yes.

Professor Dame Ottoline Leyser: I would like to pick up on this issue of the norms of science, and maybe connect the two things. It seems to me that part of the issue is that at some level we have lost sight of what science actually is.

Science is a method. It is a way of building models of the world that have both explanatory and predictive power. It is not about the ultimate quest for “Truth”. It is not about correct and incorrect; it is a progressive method for proposing, testing and rejecting or refining models of the world that have explanatory and predictive power. It moves forward extensively by being wrong and by rejecting hypotheses, and therefore being able to refine them. The way things have gone in the research system, we have developed a culture where people are rewarded for being “right” and being “exciting” in some strange way. Those things have nothing to do with science and the research method that we want to espouse.

Q7                Chair: You are suggesting that there is something that needs fixing.

Professor Dame Ottoline Leyser: Yes.

Q8                Chair: The trend has gone in the wrong direction or moved away from what science should be about.

Professor Dame Ottoline Leyser: Yes. The problem is that these norms of science, which everybody would agree to if you pushed them, are being not exactly eroded but counterweighted by the way the rewards in the research system are currently meted out.

Q9                Chair: We will come back to the incentives and perverse incentives later. Presumably, when we are talking about the conduct of researchers, it is important to distinguish deliberate act and inadvertent error, and not lump them all together. It comes back to the point about the culpability of the individuals. Is that something that you think is important?

Dr Vaganay: I agree. There are honest mistakes and there are dishonest mistakes. I am more concerned about honest mistakes, first because they are easier to identify and we have more evidence. Equally, it is very difficult to give evidence and prove that a mistake was made to mislead the reader or the user.

Q10            Chair: We talk about questionable research practices. When I looked through the examples, it seemed to me that they may involve deliberate wrongdoing, but they may also involve inadvertent error. Is that right?

Professor Bishop: Yes.

Q11            Chair: Is it a grey area, in other words? Does it all come down to the motivation of the individual?

Professor Bishop: In a sense it does, but a lot of the problems can be fixed. Some of them are down to this grey area where people have been told that they should not do something like p-hacking. You may want me to explain what that is.

Q12            Chair: I would be grateful if you could do it now, while you are on the ball.

Professor Bishop: It is best understood if you play poker or understand how cards work. That is how statisticians explain things.

Q13            Chair: Do you speak as a practitioner?

Professor Bishop: Alas no; my face is not inscrutable enough. I could say that one of you had amazing powers of telepathy and I was going to demonstrate that by giving you a pack of cards, and each of you should shuffle it and then take five cards. I would then say that one of you was going to get two pairs. If we did that once and none of you got two pairs, I would say something like, “Oh well, we’re just warming up. Let’s do it again.” We would do it again and we might have 20 deals. Then maybe the Chair would get two pairs. I would say, “That’s it; you have amazing powers of telepathy because you’ve got two pairs.”

Would you be impressed? I hope not, because if you were a card player you would know there is a one in 20 chance of that happening. If we deal enough hands, somebody is going to get two pairs. On the other hand, if I had said at the outset, “I think that Norman Lamb has amazing powers of telepathy, and if he deals five cards I am going to influence his brain for it to be two pairs,” you would be very impressed or moderately impressed if you then got two pairs.

That is the difference about doing science; it is the latter. It is the case where you make a prediction and then you use probabilities and statistics that rely on those probabilities to test that prediction. P-hacking is when somebody gathers a whole load of data and then applies those statistics wrongly to infer something, because they just focus on the one thing that looks exciting.

A lot of researchers have not had enough statistical training to understand that difference. They have had some statistical training, and often they will have a sense that it is not quite right. Somebody recently said on social media that people tend to think of it more like jay-walking when they should be thinking of it like drunk driving. You are really doing bad science if you look at loads of stuff and then just focus on the odd one that pops up, because it could very well have popped up by chance.

Q14            Chair: Part of the answer is in the training of scientists.

Professor Bishop: I looked at the submission from the Royal Statistical Society, which said that we need more statisticians. I think we do. In some disciplines there are statisticians available for consultancy, particularly in medicine. In most disciplines, there are not. It is a problem that does not affect all disciplines. In areas like humanities and physics, it is probably not an issue, but in vast swathes of social sciences and biomedical sciences we are dealing with phenomena where we rely on probabilities rather than “You always get X or you always get Y,” because we are inherently looking at rather sloppy phenomena.

Basically, people are using statistics without fully understanding what they are doing. That is extremely dangerous. We need much better statistical training, and more statisticians to deal with this issue.

Professor Dame Ottoline Leyser: I agree that statistical training is very important. One of the things that has happened as a result of exactly these anxieties is, for example, that journals are beginning to introduce rules about what you can and cannot do with your data, and how you can and cannot present them. I get very worried about those kinds of approaches. Although I agree that it would be great to have more statisticians back in the research base, whom people can consult, I do not like the idea that you abdicate your responsibility to a set of rules or A.N. Other person—as in a statistician—who tells you the right thing to do, and then you do it.

What it comes down to is the values of scientists. Your goal has to be to progress in a way that is as robust and rigorous as possible. That is what people come into science wanting to do in the first place. We just need to give them both the space and the tools to do that. I agree that early training is important, but I worry about the idea that you paste it in on top, rather than building it in right from the beginning in a much more robust and long-term way. The underpinning basis for statistical analysis of the kind we do in bioscience, for example, is really straightforward, but for the finickity detail—exactly which test, blah, blah, blah—you need some help.

Professor Bishop: If it is so straightforward, why do so many scientists engage in p-hacking?

Professor Dame Ottoline Leyser: Because we have not had that really basic idea built in from the beginning. The statistical training people get is not in the principles of statistics; it is “Here is a list of statistical tests and here is a programme that does it for you.” The training has to be about the principles and not the details. Then on top of that we need to reinstitute the basic values. The point is to reject your hypothesis, not necessarily to support it. All your data are valuable and the negative results are at least as important as the positive results, and so on and so forth.

Q15            Chair: You talk about going back to the basic values. You talked earlier about the perverse incentives and pressures on scientists these days. Is it just a question of an exhortation to return to basic values, or do you think other things need to happen and other changes need to be made, regulatory or in terms of self-governance or anything else, that would help to achieve that shift back to the important values of science?

Professor Dame Ottoline Leyser: The work that we did with the Nuffield Council on Bioethics clearly identified that people coming into the system want to do the right thing. Part of the problem is that the system is now wired so that that is not sufficiently rewarded. Is it about top-down regulations? It is about everybody in the system doing their bit to uphold the values that we all have, but there are things that have influenced the system to be now in a place where it is hyper-competitive and the rules for winning the competition are the wrong rules. Some of those things are relatively top down. There was a lot of discussion, for example, about the research excellence framework and how it is disproportionately rewarding high-profile outputs that are the kind of flashy, breakthrough-type research.

Q16            Chair: Are you saying that equally you could use the research excellence framework to shift behaviour?

Professor Dame Ottoline Leyser: Yes, we could. We have fed into the current debate quite extensively through both the Nuffield Council on Bioethics and my role at the Royal Society. There is quite a lot of support for the kinds of things we are proposing, but it does not seem to be happening, which is a bit frustrating.

Q17            Vicky Ford: I want to go back to the basic understanding of statistics and statistical principles. It is not university-level maths. It is the school A-level maths-type principle.

Professor Bishop: That is the way I explained it simply, but there are pitfalls—

Q18            Vicky Ford: Are you saying the maths that we are teaching future scientists now does not give enough of the basic principles for statisticians?

Professor Bishop: I would say it is being taught wrongly.

Q19            Vicky Ford: Let me carry on with my question. Is it at school level? Is this just a British problem?

Professor Bishop: No.

Q20            Vicky Ford: This is an issue with other statisticians and other researchers, so how can we put that block back in at school or university level for scientists?

Professor Bishop: I have a particular view on this, which is that one is taught it out of a book with a few exercises. As Ottoline was saying, it is very much, “Here is a t-test. Here is an analysis of various things.” It is divorced from your actual data at the time. You are taught about it and you might have a project where you do it. It does not give you the fundamental understanding of the underlying principles, such as I was trying to explain with the poker example.

It makes much more sense, and it is much more feasible these days, to get people to learn how to simulate datasets artificially. If you are going to run an experiment, you should be required to simulate random numbers that more or less correspond to what you thought you might want to get, and run them through analyses. You get a sense for how that analysis works and how easy it is to get things like false positive results.

We have started to do that in a little course that I am funded to do with the BBSRC—the Biotechnology and Biological Sciences Research Council. It is only baby steps, but we are experimenting with ways of trying to get people to grasp those fundamental principles. It is completely different from just getting a statistics textbook and working through it. That is not the approach that will get you there.

Dr Vaganay: I have a slightly different view. I totally agree with Professor Bishop that more and better statistical training can only help. A research project is very complex and it has a statistical dimension as well as a theoretical dimension. It has very many different dimensions. As a researcher, you have to be a good administrator, you have to be a good statistician, and you have to be good at collecting and understanding data, so there is an element of data science as well. It is unrealistic to expect all researchers to be excellent statisticians and to have an excellent understanding of the literature and the theory, as well as being good data scientists.

In my experience as a social scientist, I see that usually research teams are too homogeneous. Economists work with economists; sociologists work with sociologists; and usually statisticians work with statisticians. Perhaps a solution to the problem would be to bring in people with different backgrounds. One research project indeed needs a statistician, but it also needs someone who has an excellent understanding of the literature and someone who has data management skills. I hope we will have time to discuss the problem of reproducibility of science. This is a major issue that is very hot at the moment, and it could be solved partly—there is no magic wand—by bringing together people with different backgrounds and different skills.

Professor Dame Ottoline Leyser: There is something a bit more fundamental to do with the issue of what science is and fundamentally what statistics are. Those can be taught in school, and they should stay with people and run through.

I agree that you cannot expect everybody to be a real expert on exactly which test to use. Almost all the statistics that we do, in the biosciences anyway, are about whether two groups of data are from the same overall population or from different populations. That is basically the question. What you do in your statistics is generate a probability that they are from the same population, which Dorothy described very accurately previously. As a convention, we say that if it is unlikely—less than one in 20—it suggests they are not from the same population.

What I have just said covers 99% of the statistics that go on in my field. That is something that a large proportion of researchers do not understand. What they think they are doing is testing whether or not they are right; if their p-number is small, it means they are right, and if their p-number is big, it means they are wrong. That entire approach is deeply flawed and needs to be shifted right from the beginning in education. That means getting away from curricula that are about a list of stuff in the science curriculum, and are rather about the way science moves forward; and getting away from a curriculum in maths that is about a list of different tests that you can do in statistics but that is about the basic principles that underpin statistics. It is one of those things that, on the one hand, is frighteningly simple and, on the other hand, is totally impossible to achieve because it is a culture change issue.

Professor Bishop: I wanted to come back to your question about regulation. I noticed that the British Medical Association was very hot on that in its commentary to you. We are very concerned that we could end up with a system like we currently have for ethics regulation, which is a nightmare. I am on public record, in my blog, arguing just how damaging it is. I was pleased to see that the BMA was really supporting it. At the moment, for every single project that is done with human subjects, you fill in a big form and it is scrutinised, and paper and letters go to and fro.

My view is that it would be more sensible to treat it like a driving test. You put somebody in charge of a car; it is a big and dangerous thing that you can kill people with, but you do not get them to somehow undergo a test every time they go out for a drive. You get them trained and certified as competent and then you let them get on with it. Ethics regulation should be like that. Certainly any regulation of the other issues about integrity should be like that. It should not be something where every piece of research is scrutinised in detail.

Q21            Chair: Do you agree that we should use the research excellence framework as a sort of mechanism to incentivise the right behaviours?

Professor Bishop: The right behaviours definitely need incentivising at the highest level, but at the same time there is a problem that when there are serious concerns about malpractice, fraud or whatever, they are not adequately dealt with. I agree with most people and I hope that it is not common. When it does happen, at the moment we have a situation where the universities themselves investigate such accusations. They have a clear conflict of interest. For those purposes, it would be sensible to have an independent arbiter, who has to be independent from Government and from the universities. They would be more like an ombudsman, but would need expertise in statistics and methods so that they could examine data, and they should have teeth so that they could take steps. Obviously that would not be non-trivial to set up, but it would benefit the scientists who sometimes come under attack from people with vested interests, the people who are concerned about fraud, and the institutions themselves, who otherwise have to deal with those very messy cases.

Q22            Chair: Do you agree that there needs to be more transparency, not about the investigation itself but about the fact that an investigation has happened?

Professor Bishop: Yes. It could go into the REF that people have to report on that.

Q23            Chair: We accept that there is a degree of wrongdoing, some of it inadvertent and some of it because of inadequate training. Some of it is deliberate, and some of that is caused by perverse incentives. Is there also an issue with people who turn a blind eye? Should they be regarded as part of the problem? Should there be more of a sense of responsibility for people to report bad practice?

Professor Bishop: I would say yes.

Dr Vaganay: Conflicted organisations sometimes turn a blind eye to the quality of research. A conflicted organisation is an organisation whose primary aim is not research; for example, it could be to develop and implement policies. Government Departments commission a lot of external social research, as well as in other disciplines, but the main job of a Government Department is not to fund research or do science; it is to inform policies.

Sometimes there can be a conflict of interest within Government because external evaluators, economists or researchers are sometimes asked to evaluate a Government’s initiative, programme or intervention. The results are not always convenient to policy makers, so there is sometimes a discussion between researchers and policy makers as to how to interpret the results and whether they want to report the findings or ignore them.

Professor Dame Ottoline Leyser: I would argue that the blind eye issue goes back to the basic research culture question. If you have a research culture that is really about science and the desire to move questions forward, it welcomes challenge and discussion on the lab floor. It welcomes all those kinds of things. Then it is not about turning a blind eye. It is about, “Gosh, what is going on here?” There are constructive suggestions for everybody to improve each other and there are open data—all those kinds of really important issues. The blind eye comes from the much more closed, competitive, don’t tell anybody, don’t question anything, which is fundamentally anti-science and, I would argue, incentive driven.

Professor Bishop: One thing that is changing is that it is now becoming more accepted that there will be mistakes and that it is not the source necessarily of great shame. We all make mistakes. If data are made open, mistakes are going to be found in those data. People used to be very reluctant to make data open for that reason, but now people are beginning to realise that in fact it is kudos for you if you put your hand up and say, “An error was found; it is now going to be put right.”

Q24            Chair: And that is a good trend.

Professor Bishop: It is a good trend.

Q25            Graham Stringer: You have said so many interesting things. There will be questions on reproducibility later, but, Professor Leyser, you said basically that the search for headlines is a problem for science. Does that also mean that the way research is decided on by research councils when they look for impacts—the internal way of talking about headlines—is a fundamental problem?

Professor Dame Ottoline Leyser: Yes, I think it is. Over the years I have been doing research, some of the fundamental motives for the direction we have gone in have been quite good. There is a desire to have some kind of rigorous assessment and rigorous competition both between people to whom you give the job and grants committees to whom you give the funding. That is good, but unfortunately the basis for the competition and the things we are interested in, which are those very broad questions about the rigour of the science and the community values of the people doing it, are quite difficult to assess in some ways.

Rigour is doable, but there is quite a broad range of things that we are asking people, and teams of people, to do. They are quite difficult to assess, whereas publications and where you have published them are very easy to assess. How much money you have made in your spin-out company is very easy to assess. People have gone with the easy things on the grounds that they think they are objective and therefore are a better way to do it. That is wrong. They may well be objective, but they are not assessing what you want to assess. We have to bite the bullet and go back to assessing the much more difficult things.

The assessment of quality is qualitative, as I have said many times. It is axiomatic. We need to embrace that. The way you do high-quality assessment of that sort is to get a diverse team of people around the table—I am very glad that there are women around this table, for example—and other kinds of diversity axes to make those decisions. People put forward their opinion and you have an open discussion, very much like the way science should work, and you do your best to make a decision based on a broad basket of really important criteria, rather than saying, “This person has published in this journal and submitted this press release, which has been tweeted 5,000 times; therefore, we will give them the money.” That is not exactly where we are with the system but it is—

Professor Bishop: I think it is a bit unfair on the system to portray it that way.

Professor Dame Ottoline Leyser: It is totally unfair on the system but I am exaggerating for effect, as is a common thing to do. It is too deeply in there. There is also a very interesting point about what actually happens around the committees. For example, what actually happens on the REF assessment panels versus what people think happens? Even if none of those bad practices is happening in the committee, but the community think those things are happening, that is enough to tip the culture in the wrong direction. You see that all the time, and it is sad, because we all want to do the right thing.

Q26            Graham Stringer: Can we get on to how much we actually know? If we take Professor Bishop’s definitions of integrity of researchers and integrity of research, what do we know statistically in terms of how big a problem the integrity of researchers is?

Professor Bishop: We do not know a lot because it is all very indirect evidence. The POSTnote covered the best studies that are done on this. You may know more about it than I do; in fact, you may have the figures more at your fingertips. The methodology is that you say to people, “Have you yourself engaged in these practices?” They think, “Ooh, should I tell people?” If you then say, “Do you know anybody who is engaged in these practices?”, the numbers go up, and people often use that as the estimate. Of course, if you are somebody who is rather good at doing it and effective, it may not be obvious to anybody that things are going wrong.

We have a very poor idea of how much is actually going on, although there is a very interesting development. If we turn to the severe end, to fraud and people making up data, there are some brilliant new statistical techniques coming out. Statisticians have developed ways of identifying when datasets are too good to be true, and they have sometimes been used to identify people who are producing fraudulent data. There are methods like that, but they tend to be done on a one-off basis when somebody suspects somebody of fraud.

Q27            Graham Stringer: Do you think that will be the most reliable method of detecting fraud in the future?

Professor Bishop: I imagine that once people know about it, they will start changing what they do. It is a moving target.

Dr Vaganay: It is very difficult to find evidence of p-hacking or publication bias by looking at individual studies. To get evidence of publication bias, you need to look at large samples of studies. That is what meta-analysts and meta-researchers do. They build datasets of 100 or 200 studies that ask the same question with similar methods. Then they look at the results, and meta-analysts have found that, in the overwhelming majority of cases, the studies support the hypothesis of the researcher.

If you look at the distribution of the results or of p-values—this is research I am doing at the moment—you see that 90% of the studies that answer the same research question and use the same methodology report the same thing. That is very unusual. When you evaluate a policy, you would not expect the policies always to work and always to have beneficial effects on people. You would expect some policies to have a positive effect. You would expect some policies to have a negative effect, and some policies to have no effect. How come we see that so many studies always support the hypothesis of the researcher?

Q28            Chair: That makes you concerned, and you smell a rat.

Dr Vaganay: That makes me concerned, yes. The fact that results so overwhelmingly support the hypothesis of researchers is something we would not expect. It is not intuitive.

Q29            Graham Stringer: Are there any differences internationally when you look at research results, about the integrity of both the researcher and the research?

Professor Bishop: I do not think so.

Dr Vaganay: Not that I know of.

Professor Bishop: It is pretty universal. Researchers move around a lot. It is an international enterprise. It would be rather surprising if people were different in one place from another.

Q30            Graham Stringer: I found it rather surprising that the Royal Society said that it was not helpful to quantify the research integrity problem. Would you agree with that?

Professor Bishop: I suppose it depends on where you want to put your resources. The culture is changing quite rapidly. People are moving to realise that open and transparent science is a good thing; in the protocol for some areas of science, such as mine, I am now working in a completely different way from 10 years ago. When I do a study, I try to specify the protocol in detail in advance and register it so that people know what I am planning to do. I make all my analysis scripts public. It is a much better way of working for my science. It is slower, so it slows me up, but I am old enough for that not to matter. It actually feels much better, and a lot of people are moving in that direction.

Given that these changes are taking place, it could be argued that any estimate you try to make will be out of date by the time you have it, and that it makes more sense to focus on changing the culture in the ways we have been talking about, and changing the education of people so that these types of practice become much more normative. But it is necessary to realise that there are going to be different solutions for different fields.

We could go around trying to do more estimates of how much of a problem there is. Personally, as I say, I think it may not be the best use of our resources. You are on the Royal Society’s committee, Ottoline, aren’t you?

Professor Dame Ottoline Leyser: Yes. I agree with what Dorothy said. I would also say that there are definition issues. What do you mean by “the research integrity problem”? Is it a question of detecting what proportion of papers are actively fraudulent or what proportion of papers have some evidence of falsification, which is not quite the same thing, versus what the impact of the publication bias problem is? That is what you were talking about previously; the only data that see the light of day are data that support the hypothesis, and the data that do not support the hypothesis are not made public.

Getting meaningful and robust information about the size of the research integrity problem is very difficult and potentially very expensive. I therefore agree that it seems much more productive to work to change the culture so that those problems are less prevalent, and so that when you rerun your analysis in the future, there are just as many studies where the hypothesis is rejected as there are studies where the hypothesis is not rejected.

Q31            Graham Stringer: I have a final question. In an investigation by one of our predecessor Committees, we came across researches where huge datasets were not made publicly available and people could not check on them. Would it be your advice and recommendation that all datasets be made available in all scientific research?

Professor Bishop: When you say made available, it is a really tricky one. If people then conduct secondary data analysis, you once again get the problem of p-hacking. There is a very interesting instance from the States, where a group of people who thought that vaccines caused autism—still, after all these years—found a big dataset from some American survey. They dived into it and found that if you looked at the children who were boys, who were black, who were of a particular age range and went to a particular nursery, lo and behold there was an association between vaccination and autism. If you looked at the whole dataset, of course, there was nothing. That was classic p-hacking. The paper was published. It was subsequently retracted, but the damage was done. It is still thought to be a cover-up by the original researchers who did not publicise that amazing fact.

It varies from area to area how controversial your area is. I would say that data should be made available, but anybody who wants to look at them should have a protocol saying what they want to look at and how they plan to analyse it, rather than a free-for-all where you can just poke around and pull out the bit that happens to support your views. There are certainly a number of forces out there whose reasons for wanting to look at data are far more political than scientific.

Q32            Darren Jones: I have a question on peer review. Research published in the journals is often peer-reviewed. What role does peer review have in assessing research integrity? If there are problems with that, what do you think they might be?

Professor Bishop: We all think peer review is not brilliant but it is probably better than nothing. I am very much in favour of a model where you move the peer review process to an earlier stage, so I can say a bit about that. Peer review typically happens after the event. You do the paper, you write it up and you send it to a journal, which sends it out for peer review. At that point, the peer reviewers cannot really alter what you have done. They can only decide to accept or reject.

It is far more helpful to get peer review at the point when you have laid out what you plan to do, and then get some feedback on that. That is now becoming possible through a process called registered reports, which were introduced in some journals. You can get feedback at a point when it is useful to you. I have done this twice and it has been fabulous; I had very useful feedback of a very detailed kind before I ran the experiment.

My instinct would be to say that it is useful, but it is much more useful to have it at an earlier stage in the proceedings. That would require, again, quite a big change in culture. There are some moves to link it with some of the funders, where the funders could tie in. At the moment a proposal to do research is peer-reviewed, but the detail that you get in those proposals is not enough to give comprehensive criticism. There again, the criticism typically is not fed back to help the researcher. It tends to be just that either you get the funding or you don’t. Peer reviewers can be quite reluctant to say anything negative when they like a proposal but do not like a little bit of it.

A model where you get peer review coming in at an earlier stage of the process, and then if you do what you said you were going to do you get guaranteed publication, is a very useful one. On the one hand, it avoids all the problems that we have talked about with p-hacking and suchlike, and, on the other hand, it gets the peer review at a useful stage in the proceedings.

Professor Dame Ottoline Leyser: Dorothy knows that I need to come in at this point.

Professor Bishop: Yes; we disagree on this one.

Professor Dame Ottoline Leyser: This is majorly field specific. In my field, the notion that there is a study that you do and then publish makes no sense whatsoever. If I put in a three-year grant proposal, which is classically what is going on, the idea that I can write in that three-year grant proposal the details of all the experiments that I am proposing to do in a way that generates positive feedback across the three years—

Professor Bishop: It doesn’t have to be for that length of time.

Professor Dame Ottoline Leyser: I know it doesn’t, but it is from the point of view of the grant proposal, which is what you were talking about. I am not saying that the mechanism that Dorothy describes is not useful. I am just saying that it is not universally useful. Again, it is the notion of science as kinds of chunks. In my field, where a lot of what you are trying to do is working out the mechanism by which something happens, rather than whether the thing happens at all, you are using many different types of experiment. The turnover in the experimental system is very quick, so you change your mind about what you are doing next week based on the results from this week. It is a different kind of landscape.

That is a general point about the whole issue. The research landscape is very big, very complicated and very diverse. It is very difficult to come up with anything that will universally help, except for that underlying cultural issue. That does not mean that specific things that are super-helpful in some fields should not be encouraged and supported, but the idea that we can generalise them needs to be taken with extreme caution.

Q33            Chair: Do we develop protocols in the different areas for the best practice that should apply to that area of research, including on the issue of peer review?

Professor Dame Ottoline Leyser: We make up protocols. We make up new ways of doing experiments on a regular basis. There are assays we routinely use in my lab, where it is very clear what we have done—we publish exactly what we have done and so on—that nobody else in the world does, just because they are not using that system; they have slightly different questions, or whatever. It is just not possible to have something that is one size fits all. That is part of the reason why peer review, despite its limitations, has survived so long in the current form. At least it is flexible. We write up what we have done and we explain what it is. People look at that thing, go through it and criticise us, and comment as much as possible.

Of course it is flawed and of course it has its limitations. People use the quote: “Democracy is the worst form of government, except for all the others.” That is substantially how a lot of us feel about peer review. It is the best thing we have. It is never going to be very good at picking up fraud, because when you review something you are not reviewing it on the assumption that the person has made up the data that they are putting before you. Although you need that in the back of your mind, it is not the job of the system to spot fabrication.

Q34            Martin Whitfield: A lot of the points you have put in are very interesting, particularly about the research culture and the norms of science. The fault possibly lies at that basic foundation that we will put right in different ways for different areas. There was a point about the homogeneous nature of research teams lacking input. You have answered a lot of my queries, but I would like to ask this. Do you think there are different pressures on researchers depending on where they are in their career? Do you think there is pressure for a new researcher? Perhaps you could expand on how those pressures alter as someone moves through their career as a researcher.

Professor Bishop: It certainly has changed in my lifetime. When I came into research, one of the first things I got was a nine-year research fellowship from the Medical Research Council, which was in blocks of three years, each renewable for three years. You would do a three-year grant and then you would get reassessed in three years, but the salary was pretty much for nine years. After that I got taken up by the Wellcome Trust, with a principal research fellowship, which was, believe it or not, for 10 years, renewable for 10 years.

Those things no longer exist. The funding is much more short term and much more competitive. Junior researchers often have to write the next grant almost as soon as they have got the current grant. They can be under intense pressure to churn out publications, which affects the possibilities for doing slow, careful science.

I do not know whether Ottoline has had the same thing, but if I give talks about reproducibility to audiences, mainly early-career scientists, one of the key questions is always, “I’d love to do things the way you’re describing it, but it will wreck my career, won’t it?” If people think that, it is seriously bad news. This is where I think we have to send a signal to those people that, no, on the contrary, organisations and individuals will be rewarded for doing careful, accurate science, where they are evaluated on the quality of their ideas and methods and not on whether they get this result or that result. That would make a huge difference. At the moment, what is happening is that sometimes the people we most want to keep in science, and who want to do the careful stuff, get so demoralised that they leave, and we are left with the people who think, “Oh well, I’ll tweak it because my boss says I should tweak it.”

Professor Dame Ottoline Leyser: I agree. It is exactly what I was saying earlier. The perception in the community is that the only things for which you are rewarded are the high-profile papers. You told me off for saying that the last time, Dorothy, but that seems to me to be where we are, and that certainly came out very clearly from the Nuffield Council on Bioethics report. People think that that is the only thing that matters. In terms of where you are in your career stage, it matters right across the career, but it matters perhaps most for early career researchers who do not yet have an independent PI position with any stability. There is a strong bottleneck in the career structure, which means that many of the PhD students and the postdoctoral researchers who start in research will not be able to continue.

Some people think that that itself is a problem. I disagree, because I think we need people with those extraordinarily high-level research skills across our economy and society, sitting where you are sitting, and in industry, enhancing our absorptive capacity, and in the media, democratising science in a positive and constructive way. We need those people, and we need to support them early in their career, both to do high-quality science and to realise that they have many choices about what they can do with their lives. It is not the end of their life if they cannot become a professor, which is what a lot of them think now and is the impression that a lot of professors give them, because we are all so obsessed with what we do that we cannot imagine doing anything else.

Dr Vaganay: I got my PhD three years ago, so I consider myself an early career researcher. I gave up on an academic career quite early in the process. I was under the impression that it was not for me and that the system was somehow biased against people with my research interests and background.

First, as you are probably aware, there is a strong emphasis on quantity. You need to produce a lot of papers. I am more interested in quality. For example, when I do research, I am committed to reproducibility. I try to document absolutely every single step in my research process. That takes time, and it means that I am slower. That is a problem.

Another problem is how “impact” is defined by the scientific community. As a scientist, you are rewarded if you can demonstrate an impact on policy—social issues, health issues, and so on. I am a meta-researcher; I do research on research. There is no impact for society. Hopefully, there will be an impact on the scientific community. There is no funding for that kind of research and it is not considered a priority by Government.

The third issue is the role of disciplines. Science is still very much structured in disciplines. Meta-research—the science of science—is interdisciplinary. It is very hard to work with other people in different fields, because they say, “Where are we going to get the funding? Where are we going to publish? Is it going to help my career?” As an early career researcher, I find it difficult to commit to transparency and reproducibility, and I think that has an impact on my career.

Q35            Martin Whitfield: What follows from that is a slightly blunt question. Public or private funding? Does the source of the funding alleviate or encourage problems?

Chair: Could you keep your answers fairly tight? I am conscious of the time.

Dr Vaganay: One of the subjects I research is financial conflict of interest in research. The research that I am doing at the moment is meta-analysis of evaluations of employment programmes. I am looking at whether there is an association between funding and findings—in other words, whether an evaluation funded by the Government shows different results, on average, as the same evaluation funded by a non-conflicted organisation. I do not have results yet. We have just pre-registered the study, so I have not seen the data, or at least I have not analysed them. I am trying to play the game.

Q36            Martin Whitfield: What is your hypothesis?

Dr Vaganay: My research is based on studies. There is a huge literature in medical science showing that, yes, there is an association between funding and findings. Clinical trials funded by pharmaceutical companies, especially companies that develop the drug—the molecule—and invest millions of pounds or dollars to develop it, are more likely to support the hypothesis, interestingly. They are much more likely to support the hypothesis of the researcher or drug developer than independent studies looking at the same drug and following a similar protocol.

Q37            Vicky Ford: You may have answered some of these questions already. Was there ever a golden age of research? You talked about when you were getting a 10-year grant and then another 10-year grant. Should we try to reproduce some of that from the past, or do we need to look to a different model going forward?

Professor Bishop: I think we have to look forward.

Q38            Vicky Ford: I just want to put a few more things together. Professor Leyser, you suggested a cultural change, especially with regard to reproducibility. Has that cultural change started to happen or is there more that we can do? Is this particularly a British issue or is it an issue where we need to work with other parts of the world as well?

Professor Bishop: I do not think we can go back to a golden age, because science has changed. The world has changed. A lot more science is big data science and team science. That is another issue: how we recognise the contribution of different people in teams. We have to move away from the model of the lone genius scientist who gets all the credit, to recognise that a lot of the big problems have been solved and we now need to be doing stuff in teams to deal with the more difficult ones.

Professor Dame Ottoline Leyser: The important word is “diversity”—again. Science needs different kinds of people contributing in different kinds of ways to different kinds of projects. That needs to be reflected right across the system, in the way it is funded for example. People are endlessly saying, “We need longer-term funding,” or, “We need shorter-term funding,” but we need a diversity of funding sources that match the diversity of the research going on across the system, allowing different approaches to be adopted and rewarded.

It is difficult to run a system that is entirely driven by diversity because, as I said before, no one size fits all; there are no universals. But we can do it; it is not impossible. I am very excited by the creation of UKRI as a way to think more holistically about our research system and to try to put in place diversity of funding mechanisms and types of ways of working—a multidisciplinary individual, or whatever it is—in a way that is coherent and thinks about the whole research system. Again, it can influence the culture of the system in a positive way.

Q39            Vicky Ford: A week ago, we had the Science Minister speaking about the importance of basic science, but making sure that universities’ funding for basic science delivers value for money. If you are just going to continue to rank science by how it is published, that will continue the cycle.

Professor Dame Ottoline Leyser: Absolutely. We have to get rid of that. The place where we need more diversity is in the assessment systems, where we have a much wider range of criteria that define high-quality science. We must look across that range of criteria and be able to nuance and balance across the criteria which are the most important for the particular thing we are looking at. There is a desperate desire for one size fits all to be fair, but it is creating unfairness and distortions in the system that drive really good people out of science because it does not look like the values they want to espouse.

Q40            Vicky Ford: Are there parts of the world that are doing that particularly well—areas we might steal some ideas from?

Professor Dame Ottoline Leyser: I would say that we are doing at least as well as most places, if not better. One of the things that has been very nice about participating in this debate over the last number of years is that we are beginning to see all kinds of movement in this country. I am particularly excited by grassroots movements—different groups of junior researchers who are starting to fight. This is something that we found in the research culture report. At that time, most people were saying, “It is very bad and we don’t like it, but it is nothing to do with us. We can’t fix it. We have no power.” We tried to emphasise very strongly that everybody in the system has some power. Even if you are the lowliest of the lowly, you can do something, even in your local environment. That is beginning to take shape, and I am very excited about that. Things that we can do top down to support and accelerate that are really valuable.

Q41            Vicky Ford: That is very positive. You are seeing a cultural change, and you think Britain is leading the way in that.

Professor Dame Ottoline Leyser: It is beginning to happen.

Professor Bishop: It is really exciting. Things are changing and I do think Britain is ahead of the game. There are little pockets. There are some institutions; the Montreal Neurological Institute, as an institution, has moved over to open science. There is a centre for open science in America at Stanford. Apart from that, it is hard to think of places where as much is happening as in the UK.

Dr Vaganay: I fully agree with the idea that we need more diversity. I also think that we need more reflexivity as researchers. We must reflect on our own biases, our priorities and our values, and so on. I often say that all research is biased. My research is biased, but I do not know how, because a lot of it is subconscious. I have done research work for Government, and now I am doing meta-research looking at the quality of research sponsored by Government, so that could be a source of bias.

It is fine. I am not saying that Government should not fund research. I am not saying that pharmaceutical companies should not fund research. What I am saying is that we need to know more, and we need to know where the money comes from. Who funds the research and who are the researchers who did the study? We can use that information to appraise the credibility of results. We must think about how, as investigators, we unconsciously bias our research. Also, we must let others know what our possible conflicts of interest might be. That would be very useful.

Q42            Bill Grant: I want to touch on a subject that Dr Vaganay mentioned, which is reproducibility. A survey by Nature last year showed that 52% of researchers felt there was a “significant crisis” in reproducibility. If we were to run that again, would it be better or worse in that regard, or would you get the same result?

Dr Vaganay: That is correct. A number of high-profile studies have been published in the last four or five years in different disciplines: in psychology, cancer research and economics. On average—it varies from one discipline to another—they show that 50% of studies cannot be reproduced or replicated.

I can give a very quick explanation. Reproducibility means that, if I give you my datasets, my codes and my methodology, you should be able to get exactly the same results and findings as mine. Surprisingly, researchers find that when they try to reproduce the work of colleagues, using the same information and protocols, they find different results.

Professor Bishop: I have to correct you. That is not how it is defined in other biomedical—

Dr Vaganay: There is replicability and reproducibility; you are right.

Professor Bishop: The things you are referring to are when people are trying to do the same experiments where there are new data, in areas like psychology.

Dr Vaganay: In social sciences, we do not replicate, because it is hard to replicate a field experiment. This is something that is done, for example, in psychology where you can actually conduct exactly the same study with a different sample and a different group of people.

Q43            Bill Grant: The challenge of reproducibility varies from field to field. In certain fields, it is more challenging.

Professor Bishop: It also depends how you define exactly what you are talking about. It certainly is not as good as it should be, given that when you publish something, if you have applied the right sort of methods, statistics and so on, you should be able to get to a point where you would be reasonably confident that you would get that result again if you reran it. Quite often, that is not happening. That is really what has prompted a lot of the changes that we are talking about. This was very much investigated by the Academy of Medical Sciences in a symposium that I chaired a couple of years ago. That identified a lot of the problems and also pointed to quite a few of the solutions, which are already being taken up, particularly by the funders.

The research councils and the Wellcome Trust do not want to waste their money. The charities in particular do not want to give research funding and find that they have wasted it, so they are very strongly motivated to bring in changes. Even the pharmaceutical companies, whom everybody loves to paint as the villains, do not want to waste their money either. They have been getting quite vocal about lack of reproducibility.

The fact that the people in charge of the purse strings are now aware of the problem is one of the drivers of change. They want to make sure that the research they do is of a kind that you can build on. To me, that is the key thing. We should be able to take what has gone before and move forward on that basis, rather than keep going backwards and forwards.

Q44            Bill Grant: I am sure it is challenging to find funding for any form of new discovery research. Is it equally challenging or impossible to find money to fund reproducibility and to check on somebody else’s work? Are there funders who will fund researchers?

Professor Bishop: It has been difficult. It is beginning to change. It is beginning to be recognised, and some of the funders have started to say that they will do it. In the past, it was almost impossible to get money to show that you could find again something that had been found before. That was regarded as boring and not sufficiently ground-breaking.

Q45            Bill Grant: Is it slightly easier to secure?

Professor Bishop: It is becoming recognised that it is worth funding those things; yes.

Dr Vaganay: Can I give an example? In the Netherlands, in 2016 the NWO—the Netherlands organisation for scientific research—allocated €3 million over three years to fund replication studies. That is an example of good practice, I believe, but there are too few examples of that best practice.

Q46            Vicky Ford: There is one more question I should have asked when we were talking about cultural change. Is there anything more that we should be doing as Government, or Government institutions, to help that cultural change? It may be that more funding, or tests on reproducibility, is one of them. We are here, so what should we do?

Professor Dame Ottoline Leyser: It is difficult. We have to think carefully about the top-down stuff. Basic things like REF reform and education reform, which we talked about, are important. I think that more MPs should get their heads around what science is and what the nature of evidence is. It is a broad thing: culture change right across the piece. It is how you use evidence effectively. It is really an important thing to think about in everything we all do every day. It is not just, “There is this thing called research that is supposed to work in this way, and it is not.” This is how we think about our lives, and how we assess evidence and move forward every day.

Dr Vaganay: I would like to stress that. In my opinion, government is not a homogeneous or single entity. It includes research councils. There are things that research councils can, and should, do. For example, they could fund more replication studies. That would be an idea.

Q47            Chair: We also talked earlier about the research excellence framework.

Dr Vaganay: Yes. There is also the Higher Education Funding Council. That is the REF. There are also the Government Departments and what they can do. It is my understanding that every year the Department for Education funds between £20 million and £30 million in external social research. They have huge power, leverage and influence on the research communities because they fund a lot of research. A lot of work could be done by Government Departments to clarify their stance, and what it means, according to them, to do credible and transparent research; what steps they are ready to take.

Government Departments could do a lot as users of research. For example, they do impact assessments when they put forward a new piece of legislation. They look at the evidence that a particular reform would have a positive and beneficial effect on the population. It is not always clear how Government Departments select the evidence to come to the conclusion that policy A is better than policy B to achieve their objective.

Chair: Thank you all very much indeed for a really useful session. It seems to me that you are all pointing to some optimistic signs about a change in culture, but you are also identifying that there are some significant issues that need to be addressed, and you have pointed to some of the solutions. Thank you very much indeed for your time this morning.

Examination of witnesses

Witnesses: Dr Wager, Professor Sir Ian Diamond, Professor Walmsley and Dr Wilmshurst.

Q48            Chair: Good morning, all of you. Thank you for being here. Would you like to introduce yourselves briefly, with your name and where you are from?

Dr Wager:  I am Dr Liz Wager. I am the editor of a journal called Research Integrity and Peer Review. In my day job, I am a freelance trainer and consultant, running courses on publication and peer review. I am the former chair of COPE, the Committee on Publication Ethics.

Professor Sir Ian Diamond: I am Ian Diamond. I am principal of the University of Aberdeen, and until recently I was chair of Universities UK Research Policy Network. It is in that role that I am here this morning.

Professor Walmsley: I am Ian Walmsley, the pro vice-chancellor for research and innovation at the University of Oxford. I am also on the Russell Group working group on research integrity.

Dr Wilmshurst: I am Peter Wilmshurst, a consultant cardiologist. I am a trustee of Healthwatch, which is a whistleblowers’ organisation. I have been a member of COPE since it was founded in 1997. My hobby, so to speak, is investigating research fraud. I say it is a hobby because I have never received any payment for it. I have reported—I don’t know how many—more than 25 doctors to the GMC for research misconduct. I have also done that abroad.

Q49            Chair:  Thank you very much. As we ask questions, don’t feel that you all have to answer. We have a lot to get through. Please keep your answers as succinct as you can.

Taking into account how we define research integrity, and the difference between the integrity of the researchers and the integrity of the research, how would you describe the current state of research integrity in the UK?

Professor Sir Ian Diamond: Frankly, I think we are on an upward curve. The introduction in 2012 of the concordat was an incredibly positive move. The moves towards open research, in the broadest sense, which involves making one’s data available for people to be able to reanalyse, seem to me a wholly good thing. I think we have a change in the culture; all universities have policies on whistleblowing, for example. This is a subject that, since I first became involved in it 10 or 15 years ago, has really moved on. That is not to say that many of the issues that Ottoline Leyser raised in the previous session do not still pertain—pressure in particular—and that is something we need to engage with, but we are on an upward curve.

Dr Wager: There are a lot of good things going on, but there is a distinct lack of transparency. Many universities are doing good things but we do not always hear about it.

Professor Walmsley: I would concur that the trajectory is definitely in a positive direction. The concordat has been extremely helpful, and increasing transparency in terms of universities publishing openly what they are doing in this area has been very good.

Dr Wilmshurst: I entirely disagree. I think there is great complacency. I only have knowledge of medical research. You have to remember that much medical research is not conducted in universities; it is conducted in NHS hospitals and the concordat does not apply. Much of it is actually performed in private practice and funded by industry: for example, the MIST Trial that was funded by a medical device company—NMT Medical—where the chief investigator has appeared before the GMC. It was of course remarkable that he was appointed as chief investigator, because he had already been before the GMC six months before he was appointed to the MIST trial for falsifying research data in an unrelated trial, which was found proved.

He was—still is—the director of Headache at King’s College Hospital—not King’s College. He had approval to enrol patients in the MIST Trial at King’s College Hospital, but in fact enrolled them in his private clinic. He was found to be dishonest. He has been suspended from the medical register for only four months, despite this being his second offence. King’s College Hospital refused to do anything because they said that although he had approval there he did not enrol any patients there. After his suspension, they took him back. Earlier, someone mentioned charities being concerned about research funding. He was on the medical advisory board of Migraine Action—the migraine charity—and continued to sit on that even when he was suspended from the medical register. The GMC said that was all right. There is considerable complacency and considerable cover-up.

Q50            Chair: What about the increase in journal article retraction rates? Is it a good thing, a bad thing or a worrying trend?

Dr Wager: It is definitely a good thing. However, it is a very unreliable indicator of the actual amount of fraud. It may well be the case that we are better at clearing it up and better at detecting it.

Q51            Chair: Rather than that there is more going on.

Dr Wager: Exactly. That is not to say that there may not be more going on.

Q52            Chair: You just cannot tell.

Dr Wager: Indeed. But it is definitely a good thing.

Professor Sir Ian Diamond: I completely agree with everything that my colleague has just said. I would add that sometimes people find they have made a mistake and they are absolutely clear that they are going to retract. That is simply an honest mistake, so we need to recognise that. It is a good thing that the numbers have gone up, but it is a very inaccurate indicator.

Q53            Chair: Professor Walmsley, the Russell Group has said that it sees “no indications of large-scale or systematic problems” with research integrity. It is in the interests of universities, is it not, to downplay the seriousness and the frequency of research misconduct? Dr Wilmshurst talked about complacency. Is there a risk that universities are not being open enough in confronting that?

Professor Walmsley: Certainly I take Dr Wilmshurst’s point that universities are not the only research-performing organisations. The Russell Group, of course, is focusing on just the HEI sector. Is there a risk of complacency? There is always such a risk. Is that driven by the universities themselves being worried about it? It cuts both ways. Of course, you do not want to be necessarily seen to encourage or support people who are conducting poor research. On the other hand, your reputation relies on you having research that is of the highest quality. There are two countervailing forces, and the one where quality of research drives your reputation and drives what you are able to accomplish in research is probably the stronger.

Professor Sir Ian Diamond: It is always worth remembering that, if, for example, a grant comes from a research council, it does not go to an individual researcher. It goes to the university. The university has an enormous requirement to be worried about it, and universities are worried about it, for the very simple reason that their reputation rests on it.

Dr Wilmshurst: I hear from lots of research whistleblowers, including from the Russell Group, and not long ago from a professor who said that he had discovered that one of his PhD students had plagiarised much of her PhD from a PhD in another country. He took that to the head of school and was told to keep it quiet, and to keep it from the PhD examiners. The PhD was awarded.

I reported Banerjee to the GMC twice. He was first suspended for falsifying research at a Russell Group medical school at King’s College—not King’s College Hospital. He got back on to the medical register for three weeks, when I reported him again, and he was struck off the second time. The thing about Banerjee is that King’s College concealed his misconduct for a decade. They held an inquiry. They silenced whistleblowers. They tried to destroy most of the evidence. He did great harm to patients. That is a Russell Group university. That is actually more typical of the way people behave.

We were talking about retraction. A couple of papers by Banerjee have been retracted, but not all of the papers. When you retract someone’s work because it is fraudulent, you should consider that most of their other work is suspect. For example, there are other things that I know that Banerjee has written that are false.

Chair: Keep it tight, because of the time.

Dr Wilmshurst: There is a letter in The Lancet that is remarkable. He makes claims in the letter about research, which we now know is fraudulent and was the basis of the Master of Surgery degree that London University took 25 years to remove from him, after they were told it was fraudulent. We know that that research is fraudulent. It is still in The Lancet and it claims to support work published by a certain Andrew Wakefield. Bizarre!

Q54            Darren Jones: In the first panel this morning, Professor Dorothy Bishop said that universities have a clear conflict of interest. I think we know what your views are on that, Dr Wilmshurst. I am interested in the question of compliance versus culture. Both Professor Ians mentioned the concordat this morning. Could you speak on behalf of your members about the number of universities that have signed up to the concordat, but also your view on whether they see it as a compliance issue—that they must put things on the website and nominate some individuals to lead it—and/or whether you have seen that lead to cultural change in the universities when assessing research integrity?

Professor Sir Ian Diamond: I know that over 100 universities have signed up. It is worth saying that the number of downloads is about 40,000; it is one of the most downloaded things. I absolutely think that it is taken very seriously. I cannot speak for every university, but I know that many universities also have the concordat as part of their training for new PhD students. That is incredibly important.

It is easy to say that there is a conflict of interest, but at the end of the day, as I said earlier, universities have an enormous interest in their own reputation and maintaining it. Universities have a long history of being able to co-regulate in an effective way. It is important to say that if there is a whistleblower, for example, the investigation is not conducted by someone right next to it. The investigation will be conducted by someone from a different discipline, in a different area, in a different position who has an overview. It is in everybody’s interest to get to the bottom of the problem.

Q55            Chair: In serious cases, we heard from the earlier panel that their view was that there should be an independent person—someone from outside the institution. You are talking about someone within the same organisation.

Professor Sir Ian Diamond: Yes. I am comfortable that it is someone within the same organisation. I would be pretty uncomfortable if it was someone from the same laboratory or something like that. There is a degree of distance brought by the current system. The more you try to regulate, the more one can actually start to limit innovation in science. I do not have a problem with someone being brought in from outside, but I do not think the way the system is at the moment is broken.

Dr Wager: You talk about compliance. I looked at something very simple. Do universities have a named contact person to deal with research integrity? Before the concordat came in, in 2012, we looked and it was around a quarter. We thought that was a bit embarrassing. It is now around a third. I think that is appalling. Surely that is a cheap, simple and easy thing to do. Similarly with the annual report; it needs some guidance, but only around a quarter of universities are putting out a public report about the investigations that they have.

Q56            Chair: Professor Diamond, is that not shocking on both of those counts?

Professor Sir Ian Diamond: I am not going to go as far as to say it is shocking.

Q57            Chair: It does not indicate that they are taking it very seriously.

Professor Sir Ian Diamond: Frankly, I think they are taking it seriously. It is important to know which universities were and which were not.

Dr Wager: I will send you the data.

Professor Sir Ian Diamond: I look forward to receiving it. I also take a view that it would make sense for everybody to be expected routinely to produce those data.

Q58            Chair: What pressure does Universities UK put on universities to make sure that, say, by December, everyone has signed up to that?

Dr Wager: It has been five years now.

Professor Sir Ian Diamond: That is not a role that Universities UK can play.

Q59            Darren Jones: Whose role is that? Whose role should it be?

Professor Sir Ian Diamond: If it was to be anyone’s role, it would have to be, I imagine, UKRI. They could, for example, say, “We will not fund unless that has happened.” That is how one would have to move in that direction.

We can see it very quickly in different arenas. For example, to take gender equality and Athena SWAN, most people would agree that it was the funding agencies requiring it that made an enormous change in the culture. I can see that that would be desirable. In my opinion, it would be a good thing if all universities reported.

Dr Wilmshurst: In many cases, the people who might be expected to investigate are actually involved in the problem. Professor Eastell, research dean at Sheffield University, appeared before the GMC and was found to have put false statements in research publications. Professor Tim Peters at King’s College was found guilty of serious professional misconduct by the GMC for knowingly putting his name on false research by Banerjee. Banerjee was found guilty of serious professional misconduct. In that case, King’s did hold an investigation. Banerjee worked part time for Peters, who was associate dean at King’s College and London University. These are senior people. When the investigation was before King’s College, Banerjee had worked part time for Professor Michael Baum, so King’s College got the head of another department, who, coincidentally, happened to be Michael Baum’s brother, Harold Baum, to investigate the fraud. That is how independent those investigations are.

Q60            Stephen Metcalfe: I want to pick up on the point about Universities UK’s role in encouraging universities to report. In Universities UK’s submission, they state that they were going to write to all universities encouraging the public reporting of misconduct data. That seems slightly at odds with what you have just said.

Professor Sir Ian Diamond: No. I think what I clearly said was that Universities UK does not have a role in requiring universities—

Q61            Stephen Metcalfe: But you could show some leadership.

Professor Sir Ian Diamond: Leadership is absolutely fine. I was asked a different question, so let me answer the question that was asked, which is, should Universities UK play a leadership role and encourage? One hundred per cent. Is Universities UK playing a leadership role and encouraging? One hundred per cent.

Q62            Stephen Metcalfe: Have they written the letter that they said they were going to write?

Professor Sir Ian Diamond: I would have to check with my successor, but I would expect that to be the case.

Q63            Stephen Metcalfe: Could you report back to us? Finally, do you think that progress is rapid enough in Universities UK taking the lead?

Professor Sir Ian Diamond: I am very clear that there is a really rapid change in awareness and importance. Is it fast enough and are we there yet? No.

Q64            Darren Jones: This is a question for Universities UK and the Russell Group. Do you know how many of your members have signed up to the concordat?

Professor Walmsley: For the Russell Group, it is all of them.

Professor Sir Ian Diamond: I can provide that information. I know it is over 100, but I need to check.

Q65            Darren Jones: My second point is on independence of investigation. I am thinking about my own experience. I am a solicitor; I am regulated by the Solicitors Regulation Authority. If there was a question about my conduct or the quality of my advice, a partner in a different team from mine would not investigate me, because we work for the same organisation. It is in all of our interests to have a good outcome, in terms of our brand, so that we can get clients and funding. It seems to me that it should probably be the same case for universities. I agree with Professor Bishop that someone else, outside your institution, should do that. Equally, I am licensed to practise. If there is a question of fraudulent or dishonest behaviour, I get struck off and cannot practise. I know that is the case in medicine and dentistry, but is it the case in other science research areas? If it is not, should it be?

Professor Walmsley: There are different aspects. As Professor Leyser said, it is a diverse spectrum of different sorts of activities. An example would be that, if you are doing research on animals, you need a licence. If you are found to be not in compliance with the ethical protocols, you lose that licence and there may be further consequences, depending on the nature of the transgression. Of course, in other areas there is no licence to undertake research. That is why it is an important cultural issue for us to make sure that the issues of integrity are properly embedded in all practices, from research students through to research staff and to academics.

Dr Wager: One thing that particularly concerns me is the secrecy between universities. Somebody can be fired from a university and let go quietly, perhaps with a glowing reference. There is the case of Ahluwalia that I referred to in my written evidence. He started out at Cambridge and was fired, and ended up at Imperial. We do not know about that, but he seems to have done some dodgy things there. He ended up at University College London. That is where he got caught, but he had moved on to the University of East London by the time he was caught. That was four very reputable British universities not sharing information. I think the idea of some kind of licence or public list is a good one. They do it in Pakistan; if you get caught for plagiarism, there is a public website I can look at to find out if I want to employ you or not. I think that is a real area of concern.

Q66            Chair: Dr Wager, could it ever be right to have a confidentiality clause in a compromise agreement terminating someone’s employment, so that they can disappear quietly?

Dr Wager: Personally I think not, because the journals then cannot get the information. They cannot inform readers that research is flawed. Future employers cannot get the information. I can understand the motives, but we need much more transparency over that.

Q67            Chair: It seems to me that it could be a change in iteration to the concordat, that that sort of secrecy should be completely outlawed. Would you agree with that, Professor Diamond? Can it ever be right to have a confidentiality clause about the reasons for your disappearance?

Dr Wager: It affects employment law as well. It is quite complex.

Professor Sir Ian Diamond: I defer to your colleague to your left, who is a professional in this area.

Q68            Chair: So am I, actually.

Professor Sir Ian Diamond: I am a humble statistician, and therefore I am always very nervous about commenting without taking legal advice.

Q69            Chair: But you would not have a problem with that: to exclude the idea of a privacy/secrecy agreement in a termination of employment.

Professor Sir Ian Diamond: In principle, but I absolutely do not know the legal implications.

Q70            Vicky Ford: I want to go back to what Ian said about UKRI potentially making it a condition of giving grants that the university was in some way compliant with the concordat. You also suggested that the one-size-fits-all strategy for compliance might not be the best way forward. Are there times when the university would have a research integrity policy but not necessarily an individual officer responsible? I can see why, if you have a very diverse university, having an individual as the named person might not be the best policy. Are you saying that each university should have its own policy?

Professor Sir Ian Diamond: To say there is one such thing as a university in the UK would be wrong. One of the absolutely brilliant things about UK higher education is its diversity, and we should celebrate that. Therefore, it seems to me that it is difficult to say that there is a one size fits all. Ian Walmsley leads one of the most diverse institutions, and it seems to me that it may well be that you would have a number of people. The critical thing, where I come from, is that everybody knows who that person is and everybody knows how to contact that person, and that there is a very clear and open whistleblowing procedure. Those sorts of things seem to me to be absolutely required.

Dr Wager: It is a point of contact.

Professor Sir Ian Diamond: Then you need diversity in the system. It just needs to work.

Q71            Vicky Ford: But not just one individual.

Dr Wager: It is a contact point.

Professor Walmsley: I would agree with that, but there are several different aspects, as Ian said. There is the ownership of the culture as a whole; at Oxford, for example, that is me as chair of the research committee. It is a regular agenda item on the university research committee. It is surfaced at a very high level and the tone from the top is absolutely about high-quality and high-integrity research.

There is a separate procedure with contact individuals within research services who can help support investigations. Then there is a person, the registrar, who is the particular point person for formal investigations. It is an issue of placing responsibility and accountability where it is most appropriate for the institution.

Dr Wilmshurst: There are clear examples where that is not true. Professor Peter Richards signed a confidentiality agreement to conceal the misconduct of Clive Handler from the GMC. I reported Handler to the GMC and he was suspended. Richards was the dean at St Mary’s. He was pro rector of Imperial. He was the chair of the Council of UK Medical Schools and Faculties. Some of the most senior people were agreeing illegally to conceal serious misconduct. That is the problem. Whatever they may say, the reality is entirely different.

Q72            Bill Grant: Dr Wilmshurst, you touched on NHS research and how poor outcomes impacted on patients. At the same time, you said that there was no involvement with the concordat that has seen some improvements in universities. For my benefit, and for others, what checks and balances, or governance, are there in NHS medical research?

Dr Wilmshurst: In most hospitals, there are directors of research and development who are responsible. Of course, we are talking about human research. Ethics committees are nominally responsible, and we were told earlier about the difficulty of getting things through ethics committees. In fact, they do not do a very good job. I can think of a trial that I was involved in where one of the principal investigators enrolled patients without fulfilling the proper ethics approval procedure. The ethics committee allowed the protocol to be changed six months after the data were in. Of course, the manufacturer of the device could alter the outcome if you changed the protocol, the analysis. In fact, they were criticised at the GMC, when one of the investigators appeared before it and was found guilty of misconduct, for blindly accepting statements given to them by a company employed by the manufacturers, the CRO, the Clinical Research Organisation. They are often very amateurish.

Professor Sir Ian Diamond: In universities right across the United Kingdom, the use of ethics committees has been something where, again, in the past decades there have been increasing standards. I have to disagree with the idea that university ethics committees are amateurish. They put in an enormous amount of time. There is a lot of training going in. At the end of the day there is a judgment, but it is made around pretty strong protocols. I do not know whether Ian feels the same.

Professor Walmsley: I entirely agree. I would add that most, if not all, of those ethics committees have external members.

Dr Wilmshurst: I was speaking about NHS ethics committees, not universities.

Q73            Vicky Ford: May I come back on this point, because you have used some very strong language? You clearly described a couple of cases that sound very worrying. You went on to say that this is typical, and that ethics committees are often very amateurish. I contend that those are quite sweeping statements. I suspect we should have a whole panel about the NHS and medical research, because I would not want to leave the constituents I represent feeling concerned, as patients, going into medical trials. We have tens of thousands of medical research cases going through, and I contend that it is a small amount that have this sort of issue. Let us be careful about the language we choose. These are very serious cases, but we must be very aware that, if you have evidence for “often” or “typical”, we would like to hear how it is justified.

Dr Wilmshurst: I would respond by saying that I think this is at the senior level. In the National Research Ethics Service, for example, I can tell you of a researcher who was kicked out of one trial and told he was not fit to be a principal investigator in a clinical trial. Six months later, he was back before a different MREC—multicentre research ethics committee—as the chief investigator. The National Research Ethics Service and COREC had no way of knowing that an entirely different MREC, the Yorkshire and north-east, had said six months earlier that he was not fit to be a principal investigator when he applied in the midlands to be the chief investigator.

Q74            Chair: Dr Wilmshurst, you are identifying a number of specific cases where serious things have gone wrong. We have to be careful about generalising. I think that is the point that Vicky is making. Presumably, you would accept that, would you?

Dr Wilmshurst: I accept that, yes, but I am just going on my experience doing research for the last 35 years, and from speaking to people on ethics committees.

Dr Wager: Commercial research and the development of drugs is actually much more heavily scrutinised and audited than academic research. My background is the pharmaceutical industry, and the level of fraud and fraudulent data in industry research is probably a lot lower because, as I say, there is a lot more regulation involved.

Q75            Chair: Dr Wager, in medical research are ethics committees responsible just for safety and consent issues, or do they also have a role in ensuring good research in itself?

Dr Wager: Do they or should they?

Q76            Chair: Both. Quickly.

Dr Wager: Currently, they really only look at the beginning. They look at the design and they look at the protection of patients. I would love them to get more involved in the later stages—the conduct and then the reporting of the research. Currently though, they are under-resourced to do that and generally do not.

Q77            Graham Stringer: I have two questions. Professor Diamond, I want to take you right back to the point made at the beginning about research datasets being generally available. You heard Professor Bishop’s evidence earlier, I assume.

Professor Sir Ian Diamond: I did not hear all of Professor Bishop’s evidence.

Q78            Graham Stringer: I asked Professor Bishop whether datasets should be made available as a regular item, and she said, “Yes,” but there was a “but”. The “but” was that it depends on the intention of anybody wishing to interrogate the datasets. She used a case where people were misapplying statistics and trying to draw the conclusion that black people suffered more from autism. I think that was the example she gave. Would you put the same restriction on that?

Professor Sir Ian Diamond: I did not hear that part of Professor Bishop’s evidence. I come from a pretty strong view that actually data should be available, but that also requires the metadata around them—some people call them para-data: how they were collected and what the key elements are. Indeed, in the social sciences in the United Kingdom, certainly for the past 50 or 60 years, data routinely collected under the auspices of the Economic and Social Research Council have had to be deposited at what used to be called the Data Archive but is now called the Data Service, at the University of Essex, and made available for secondary analysis. I have personally undertaken research using such data on many occasions. It is almost a culture of social science. One of the first things one does when one gets the data is to try to replicate any results that have come. I personally think that it should be the norm that data are made available, as long as you know how they were collected.

Q79            Graham Stringer: Going back to research integrity, can you tell the Committee how many researchers have been stopped doing research at the University of Oxford and the University of Aberdeen for research misconduct?

Chair: In the last five years or so.

Graham Stringer: Yes.

Professor Sir Ian Diamond: I will be very happy to send you a note pretty quickly responding to that, to give you the exact result. It will be tiny.

Q80            Graham Stringer: Can you just give us a ballpark figure? To take the Chair’s point, over a five-year period, would it be one a year, 10 a year or none a year?

Professor Sir Ian Diamond: At the University of Aberdeen, my expectation is nought in recent years.

Q81            Chair: Is it something we should worry about?

Professor Sir Ian Diamond: No, I do not think we should worry about it.

Q82            Chair: You think the research has been perfect, with no wrongdoing in five years.

Professor Sir Ian Diamond: Chair, there is no such thing in my opinion as perfect research. One does the experiment, the analysis or the interviews to the best of one’s ability, and one tries to conduct things that are reproducible. I would worry very much about saying that things are perfect. I come back to what Professor Walmsley said. We need a culture. I believe that the University of Aberdeen has a culture whereby integrity is incredibly clear. It is important that there is training, and a clear knowledge of how, if one suspected that something was going on, one would do something about it. There is clear evidence across the UK, at times, of research misconduct; no question. Is it a very common thing? No.

Q83            Chair: Professor Walmsley, can you answer Graham’s question?

Professor Walmsley: Typically, we have in the order of half a dozen to 10 cases that go to inquiry within the university.

Q84            Graham Stringer: Is that per year?

Professor Walmsley: That is per year, yes. It is on our website. You can see our response to the concordat. We publish those data. Typically, in the order of one or two of those are found to have had some significant case where action was needed. As far as I am aware, in the past five years, probably one researcher has been restricted from undertaking certain kinds of research. That is the sort of order we anticipate.

Q85            Stephanie Peacock: You have touched on this point at different parts of the session. Could you outline how universities respond to allegations? You have answered a lot about that, but how long does it typically take?

Professor Sir Ian Diamond: It very much depends. If I say that one size does not fit all, I am not trying to be at all evasive. I would be very disappointed if I were to learn that things were not taken very seriously and very quickly, so that as soon as there was any knowledge, an investigation would be put in place very quickly, with all due speed, to allow proper time for both sides to be heard. Let me be very clear about the urgency. One wants to clear it up. As Professor Walmsley said, it could be that there is something important to look at but it may not be found to be the problem. If it is the problem, you want to step on it very quickly. In my university, it would be looked at very quickly by a senior person.

Professor Walmsley: I concur. Some issues can be resolved very rapidly at local level. Others, when they get to a formal inquiry at university level, may take a little longer to gather all the evidence and make the assessments.

Q86            Stephanie Peacock: What processes are in place for whistleblowers? Are people protected if they come forward with information?

Professor Walmsley: Yes.

Professor Sir Ian Diamond: Absolutely.

Dr Wilmshurst: None of them.

Q87            Stephanie Peacock: Those were simple answers. You touched on this earlier: is there a public record of those found guilty of public misconduct?

Dr Wager: No, absolutely nothing. You were saying that Oxford has its annual report. Only a quarter of British universities post even a very high headline figure about this. I feel that is something very concrete in the concordat that needed some more guidance. It needed a bit of clarity about what we meant about investigation and so on, but it is a very low cost and a very simple thing that could have been done that was not done. It is very useful to encourage trust, and to encourage universities to see what other people are doing, share best practice and so on.

Q88            Stephanie Peacock: Do you support that?

Professor Sir Ian Diamond: I support that too.

Professor Walmsley: So do I.

Dr Wilmshurst: My experiences of being telephoned by whistleblowers, including professors who feel they cannot speak out or are being silenced by their universities, are quite clear. In fact, I have a letter given to me by a whistleblower. It is about the Royal Brompton. He received a letter from the chairman of a past board of governors advising him to keep quiet. He has it in writing from the chairman of the board of governors. I have a copy. His career will suffer “unless you drop this matter.” That is in writing, and you can have a copy of the letter.

Dr Wager: You might be calling the UK Research Integrity Office for a later session. They run a confidential helpline. They will give you figures about the number of people who contact them from British universities. It is a little out of date now, but they published a mechanism or advice to universities about how misconduct cases should be handled. I suggest you might ask them for some good information.

Q89            Stephanie Peacock: According to the Nuffield Council on Bioethics, only 60% of researchers think that different initiatives to protect integrity are a good thing. Do you think that statistic of 60% is disappointing?

Dr Wager: I find that very odd and very strange.

Professor Sir Ian Diamond: As a social scientist who works on surveys, in order to answer that question I would need to know what the question was and what the other questions were, and I would want to know what was asked of the 40% who said that it was not.

Dr Wager: When I run training for researchers all around the world, generally the great majority want to do the right thing. They want to find the truth. There are a small number who do not. I find that response odd. Maybe they were thinking about red tape, about bureaucracy and about extra layers of regulation, to which researchers are generally rather allergic.

Q90            Chair: Professor Diamond, the Universities UK submission said, “Since 2012, significant progress has been made in delivering greater coordination between the critical partners in supporting research integrity, and the Concordat to support research integrity has been an effective stimulus to this end.” What is the evidence that it has had that great effect? On the face of it, I cannot see it.

Professor Sir Ian Diamond: The fact of the matter is that, right across the university sector, research integrity is something that is seen at a high level and at a low level in a way that it simply was not a decade ago. Those of us who were at the Keele meeting about this in 2008 were almost at day one. We have moved an enormous way and the research concordat was effective. Is there more to do? That is why Universities UK has a research integrity forum, to make sure that we share best practice. I agree that it is important that we share best practice and continue to improve. The concordat is joined up between the major funders, the universities and the funding council in a helpful way. Again, we are moving in a direction that we would not have taken without it.

Q91            Chair: Universities UK has also explained that compliance with the concordat is a prerequisite for receiving funds from the funding councils and that this means that “institutions can face sanctions should they be shown to be failing to meet the commitments.” Have any universities or institutions faced sanctions?

Professor Sir Ian Diamond: Not that I am aware.

Q92            Chair: Which again causes me concern. If we have gone through several years since the concordat came into force, and there has not been any institution that has faced sanction, it suggests that it is not perhaps working as effectively as you indicate.

Dr Wager: The concordat is an excellent document. It came at a time when internationally there was increased attention on research integrity. We cannot say what the effects of the concordat are. From my experience, there really is limited evidence that it has improved things.

Q93            Chair: From my experience as a Minister with a concordat, we felt that we had to drive it; we had to show leadership, encouraging and requiring everyone to sign up, to have the officer identified and to publish the annual report. It just feels like there is complacency: 26% of universities publishing an annual report. That is hopeless, isn’t it?

Professor Walmsley: It might be helpful to think of the following. For the Russell Group, all 24 institutions have signed up for that.

Q94            Chair: And they publish their annual report.

Professor Walmsley: I am not sure if it is all 24 yet.

Q95            Chair: But you would say that they all should.

Professor Walmsley: They all should; that is absolutely right. It is incumbent on an entity like the Russell Group to show that sort of leadership. Of course, that is the majority of research capacity in the UK as well. We should also recognise that the majority of that is something where we are really driving hard.

Dr Wager: But it is probably the less experienced universities that may need more help. That is another point.

Q96            Chair: My plea is for leadership from Universities UK to drive this, not just to sit back and accept it.

Professor Sir Ian Diamond: Can I just disagree with the word “hopeless”?

Dr Wager: I like it.

Q97            Chair: But 26% after four years is simply not good enough.

Professor Sir Ian Diamond: I am prepared to take that point, but at the same time we need to work very hard to encourage people to move in that direction.

Q98            Chair: Surely, you should just be saying, “We have to do this. We cannot tolerate a lack of transparency.”

Professor Sir Ian Diamond: The fact of the matter is that universities are autonomous institutions.

Q99            Chair: But you could show leadership.

Professor Sir Ian Diamond: I would argue that Universities UK is showing leadership. It is asking people to do things. It is not in a position—

Q100       Chair: But it is not working.

Professor Sir Ian Diamond: It is not for Universities UK to require. That is not a role that Universities UK plays.

Q101       Stephen Metcalfe: Picking up on the point about the Russell Group, you hoped that they were all doing it, yet when the BBC tried to obtain information earlier this year from the Russell Group universities, it was unsuccessful. It could not get information in every case. Do you think it is reasonable for a university to withhold that information? Why might they not want to co-operate with informing the BBC that they are open and honest about this? Maybe honest is too strong.

Professor Walmsley: Universities should be transparent about it. Of course, I do not know what questions the BBC asked. The notion that you publish a statement about how you are complying with the concordat, including the statistics of cases that you have in your institution, seems to me a very reasonable sort of thing to do. That is where UUK and the Russell Group can show that sort of leadership, to make sure that it continually happens. In that case, the BBC could just go to the website and get the information.

Dr Wager: They don’t need freedom of information.

Q102       Stephen Metcalfe: What should we read into the fact that this information is not readily available?

Professor Walmsley: Perhaps the culture has not yet moved on as far as it needs to move on.

Q103       Vicky Ford: I want to go back to Professor Ian Diamond and the UUK. Some of the universities that are not Russell Group universities may not be doing a huge amount of research. I can understand that a one-size-fits­‑all approach that they have to produce an annual report on research integrity might not be appropriate for them. Would you say that maybe a multiannual report would be more sensible, or is there a way we can share best practice a bit better? Then there is a question you missed earlier: is there another country doing this better? We heard from the previous panel that in the UK there is a step change, coming from the researchers themselves in many disciplines. Are there other countries that are doing better?

Professor Sir Ian Diamond: I am not aware of other countries that are doing better. Denmark and the United States are sometimes held up, but I am not clear that their processes are much better. I take a personal view that it should be quite easy for all universities to produce an annual report and for it to be on the website. If you are at a university whose mission is largely towards teaching, and the amount of research is relatively small compared with a university like Oxford, it will not take terribly long to write the report. Therefore, it seems to me that it is still a good thing to do.

Q104       Chair: Dr Wilmshurst, I wanted to give you the chance to comment on the speed of response to research integrity problems. Do you want to add anything on that?

Dr Wilmshurst: It is far too slow. The conflicts of interest mean that universities cannot self-regulate. I was going to clarify something about the ethics committee, but it might be too late.

Chair: Make the point.

Dr Wilmshurst: The point I was going to make was that ethics committees are reasonable at dealing with applications for ethics approvals. But if you ask them, they are not trained at all to deal with allegations of research misconduct. They say that publicly; there is no doubt. In that respect, when you go to them as the ethical body, they say, “We cannot deal with this. It is nothing to do with us. Our job is just to approve it.”

Dr Wager: But broader things like non-publication do not take much qualification to look at. I agree that they should not be there for investigation. Can I pick up on the point about whether other countries are doing it better? One model that has a lot of merit is the Australian system, where the first response lies with the university. That is the prime responsibility to investigate, and it is an important principle. We do not have some big expensive body trying to do the investigations. However, in the last four or five years, they have set up an independent but national committee on research integrity that acts like a court of appeal. If the university has not done a good job, if people are unhappy or it is too slow, or to collate best practice and give advice, they have that extra layer. That is something we really lack in the UK.

We have the UK Research Integrity Office, which has no official funding or status and is purely advisory to individuals. We do not have anything that universities, individuals or whistleblowers can turn to if a university does not do a good job. Often they investigate well, but there are cases when they do not investigate well, and that is when the system really falls down, because we just say, “Well, they are autonomous and secretive,” and there the matter ends.

Professor Sir Ian Diamond: But that is where the helpline for the UKRIO comes in.

Dr Wager: But it does not go any further. I like the Australian system because they have a national committee that has more teeth. UKRIO does not even have any funding, let alone teeth.

Dr Wilmshurst: Someone asked about timing. I should say that I only know about medical research. Of course, you can appeal to the GMC if the university will not do anything, but they are even slower. The last case that got dealt with by the GMC I referred in September 2008, and it came to a three-month hearing starting in November 2014, so that was over six years.

Q105       Martin Whitfield: I want to take up Stephen’s point as a matter of interest. Who do you think should be held accountable for research integrity? Is it the university? Is it the individual? Is it the team? Who should take the blame?

Dr Wager: You should never be blamed for integrity. It depends what you mean. Who should be blamed for misconduct? At the heart of it, it is usually an individual but they are working within the system. There is a big difference. Research integrity covers a lot more than just preventing misconduct. Institutions should take responsibility for the whole of research integrity, which includes carefulness, reproducibility and not doing sloppy research, as well as properly investigating the relatively rare cases of deliberate fraud and misconduct. We have to use the terms carefully. Integrity is sometimes seen as a euphemism for lack of misconduct, whereas actually it is broader than that.

Professor Walmsley: I concur. The key issue about culture is that everybody is responsible. A parallel example might be around health and safety. There, everybody is responsible, but there is a hierarchy where it is managed, and protocols.

Dr Wilmshurst: In fact, if you look at some cases it is quite clear that people have known for years that research misconduct was going on. Bob Slutsky had 175 publications retracted. People knew for five years, and people were putting their names on his publications knowing that they were false. I even had the wife of one of his co-authors contact me because she was worried that, when Slutsky was exposed, her husband would be implicated. In fact, all those people get off. They say it is just one person, but in many of the cases, it is very difficult to see how you could not know that someone was falsifying large numbers of publications.

At present, we have a man as president of the European Society of Cardiology who published over 500 publications with Don Poldermans. Don Poldermans falsified research. It is hard to see how people do not know. They must know in many cases.

Q106       Martin Whitfield: On the basis of that, and particularly from the universities’ point of view, do you think it is time to revisit the concordat, redraft it, give examples and then perhaps try to roll it out with more enthusiasm over, shall we say, 18 months rather than five or six years?

Dr Wager: Absolutely. I would love to see that.

Professor Sir Ian Diamond: It is probably always time after a couple of years to revisit and say, “How are we doing?”

Q107       Chair: Could you take that back?

Professor Sir Ian Diamond: I will take that back; I am very happy to.

Q108       Chair: It seems to me that there is a lot that points towards the need for an updating of the concordat. I can see that it is a good thing, but, as Martin says, it needs to be driven with more enthusiasm.

Professor Sir Ian Diamond: I am very happy to write to the new chair of the Research Policy Network and urge a revisit of the concordat.

Dr Wager: And with mechanisms that really encourage it, as you say. For example, I was appointed to a working group or advisory group for the concordat. It never met.

Q109       Chair: Professor Diamond, could I call that hopeless?

Professor Sir Ian Diamond: I will let you off on that one. To respond very quickly—I have said it five times and I know I am a broken record—Universities UK cannot require.

Q110       Chair: We accept that, but leadership is another thing.

Professor Sir Ian Diamond: Leadership is fine.

Q111       Chair: I have a final question, on criminal offence. Given that the consequences of falsification in carrying out research can be very serious, is there a case for making it a criminal offence?

Professor Sir Ian Diamond: It already can be a criminal offence. One needs to investigate, and in a position where one believes something should be taken to law, it should be.

Dr Wager: There are some downsides. The Scandinavian system is very legally based. That means that the day the law was written was when they invented research misconduct. If you committed misconduct the week before that, it did not count because you cannot have retrospective legislation. Similarly, it does not always help journals sorting things out, if the law comes down heavily. I have mixed feelings about that. Actually, I think there are perhaps better ways of dealing with it.

Dr Wilmshurst: I am in favour of bringing in a criminal offence. It is important because the concordat does not cover most medical research in the UK.

Q112       Chair: It could be extended. We could bring more bodies into the remit of an updated concordat.

Dr Wager: It uses the language “research employers.” It does not say universities. It is specifically inclusive.

Q113       Chair: I have another question, and please can we have very quick answers? You heard the first panel talk about the research excellence framework and whether it could be used as a mechanism to incentivise good behaviour. Do you agree or disagree?

Dr Wager: It certainly should not be a perverse incentive. We need to look carefully to make sure that it is not encouraging the wrong behaviour.

Professor Sir Ian Diamond: There are two things. It has been useful in bringing the concordat to the fore. The other point is that under the new changes for the next research excellence framework, following the Stern report, it is now not the case that everybody who is submitted has to have four publications. That is incredibly important, because the race for four publications put on enormous amounts of pressure, and it could be career ending not to be in. Now that it is between one and six, or whatever the final agreement is, it seems to be a very positive move.

Q114       Chair: Do you have any views, Professor Walmsley, about whether the research excellence framework could be used more effectively?

Professor Walmsley: Plausibly. It certainly had a big effect in surfacing and helping universities articulate what impact they were having. I do not see it, from the perspective of the Russell Group and certainly from Oxford, as having any perverse incentives. Four publications over seven or eight years is not, in my view, an inordinate pressure to deliver in a productive research environment.

Dr Wilmshurst: You have to have the right incentives, but in much of medical research the incentives are financial, from industry. That is an entirely different thing, because you can become very rich if you do the right research and get the right answers for the right companies.

Chair: Thank you all very much indeed for your evidence. It is enormously appreciated. The session is at an end.