Written Evidence Submitted by Professor Dorothy Bishop, Professor of Developmental Neuropsychology, University of Oxford

(RRE0013)

Introduction

I am an academic researcher with a longstanding interest in open and reproducible science. I am Professor of Developmental Neuropsychology  at the University of Oxford, and a Fellow of the British Academy, Fellow of the Royal Society, and Fellow of the Academy of Medical Sciences. In 2015 I chaired a 2-day symposium organised by the Academy of Medical Sciences on Replicability and Reproducibility of Biomedical Science[1].  Subsequently, with Prof Chris Chambers from University of Cardiff and Prof Marcus Munafò from University of Bristol, I ran an annual 5-day course for 4 years on Advanced Methods in Reproducible Science for early-career researchers, funded by the Biological and Biotechnology Research Council[2]. Subsequently, I played a key role in setting up a cross-disciplinary group known as Reproducible Research Oxford (RROx), with support from the University, which aims to promote open and reproducible scholarship across all subject areas. RROx[3] is a satellite organisation under the UK Reproducibility Network, which is submitting a separate response to this consultation.

Breadth of the reproducibility crisis and what research areas it is most prevalent in

My knowledge is mainly focused on psychology, neuroscience and biomedical sciences. The problems in psychology have been recognised and documented for some years[4], with surveys indicating a high level of problems relating to use and interpretation of statistical methods. In contrast, the clinical trials field recognised the potential for problems early on - with researchers such as Iain Chalmers and Doug Altman in Oxford stressing the need for procedures for registering and reporting trials to ensure their integrity[5]. However, as became evident at the Academy of Medical Sciences symposium, pre-clinical biomedical sciences have problems that are at least as serious as those in psychology but are less well-recognised.  These relate not only to misuse and misinterpretation of statistical methods, but also to inadequately specified lab protocols[6], and, in some cases, contamination of reagents[7]. Some of the social sciences have issues with replication and reproducibility[8], though economics recognised they had a problem about 10 years ago and their major journals got on top of it, by setting specific criteria for papers to meet, which include the need to make data and analysis scripts openly available[9].

Issues in academia that have led to the reproducibility crisis

A succinct summary is shown in the diagram below. In brief, there is no single cause, but a myriad of contributory factors, some of which may be regarded as 'top down' (e.g. incentive structures for researchers) and others as 'bottom up' (e.g. lack of training of researchers).   The boxes on the right show potential solutions, colour-coded to show which problem they target.

Roles in addressing the reproducibility crisis

Research funders have a key role to play, because without funding, most researchers cannot operate, and so there is a strong incentive for researchers to adhere to whatever guidelines or requirements funders adopt. It is greatly in the interests of the funders to take steps to improve reproducibility of the research they support, to avoid wasting resources. Specific suggestions are given in the next section.

The current situation, which is wasteful and impedes science, will not improve unless research institutions and groups adopt different standards, moving away from an emphasis on publishing exciting-sounding results in glamour journals to requiring that all research coming from their institution or group is of the highest standard, both scientifically and ethically. Institutions have a role to play in updating the training of their researchers, but the situation is unlikely to improve unless the incentive structures alter, so that work is valued for its scholarship, credibility and transparency, rather than where it is published. 

Individual researchers need to recognise that the purpose of doing research and publishing is to find things out and communicate them. That sounds self-evident, but sadly, we live in a world where most researchers see publications as symbols of success.  But asking researchers to change their attitudes is futile unless the incentive structure changes.  In many institutions/disciplines if a colleague comes up in the coffee area and says "I've just had my paper accepted", the first question asked is not "Oh, what is it about?" but "Oh, which journal?".  Funding is also seen as a marker of prestige, with many researchers regarding the amount of grant income as an index of their self-worth. These attitudes are corrosive to science, but have been engendered by institutions, whose hiring and firing criteria are often explicitly tied to publications in "high impact" journals and amount of grant income.  Someone who published important and useful research with minimal funding would, these days, be under threat of redundancy. Some of the Nobel laureates of the past would be unemployable - or not "REFable", as current parlance has it.

Another issue concerns training. Many fields are fast-moving and some of the problems that we currently have are due to researchers using ever more complex methods, but without a detailed understanding of them. Working in new ways, with open reproducible methods and data is slower than current scientific practice, and this makes many researchers argue that they don't have time to work that way - especially if they are young without an established post. This should change, but it comes back again to incentives, as currently, many early-career researchers regard it as equivalent to career suicide to work reproducibly, because this takes more time and exciting results are less likely if questionable research practices are shunned.

Publishers are commercial operations, and their concern for research quality is driven by the fact that this affects their brand value. Although they can talk the talk about open science, in general they have been obstructive towards measures to make the literature more accessible, as this would affect their profits. They are very variable in how they respond when problems are discovered with papers published in their journals. People who have acted as whistleblowers routinely report that they are ignored or have their concerns kicked into the long grass[10]. Publishers in turn point out that some criticisms are not valid, and others require the involvement of lawyers, given that authors may sue if their paper is publicly criticised, and so they cannot just retract work when concerns are raised. That is fair comment, but currently the balance is in favour of the person who publishes erroneous work, rather than in favour of readers, who deserve to know when the published work contains flaws.

Governments as the source of funding to UKRI could potentially could impose requirements that grantees would need to meet in order to obtain funding - see below.

Policies or schemes that could have a positive impact on academia’s approach to reproducible research

Some actions are implicit in what I have written in the section above and indeed in the diagram showing possible solutions: requirements for open, transparent data, protocols and materials, and pre-registration of protocols would help identify which research is reproducible. We need changes to the criteria for awarding grants and hiring/firing to move away from emphasis on where you publish, and how much, to whether the research addresses an important question and is credible. I suspect other respondents will document such options, so I will take the opportunity to make some more radical proposals that I think could make a difference.

A. Actions by Government.

Organisation to act as "honest broker" to investigate whistleblower complaints.

It is customary to make a sharp distinction between research that is not reproducible because of fraud, versus problems due to other reasons, such as questionable research practices. Nevertheless, it could be argued that both fraud and questionable practices have adverse effects, and a common mechanism to detect and deal with research error, arising from any cause, would improve science. Currently, we lack mechanisms to deal with cases where erroneous research gets into the academic literature. And because publications are what make researchers successful, there is remarkable reluctance to act when errors are discovered - as if it is a socially unacceptable thing to comment on.

One solution would be to create an impartial, trusted, government-sponsored body that acts as an "honest broker" between whistleblowers and those they blow the whistle on, which is independent of research institutions, and employs experts who are skilled at data-sleuthing and capable of evaluating quality of data and analysis in a range of scientific areas. The body would also have the power to impose sanctions against individual researchers or institutions who failed to act when serious errors were found in published research - either because of sloppiness or fraud. The intentions of the researchers are less important than the consequences. We all make errors: a good scientist is mortified when they are discovered and rapidly retracts or corrects the work. Unfortunately, there are too many cases where this does not occur.

I will illustrate with a recent case highlighted on Twitter[11]. This concerns an Australian microbiologist, Chris Greening, but the issues described are universal. Greening related "the worst experience I had in academia. A combination of scientific misconduct, peer review interference, bullying, abuse of power, and lack of credit in one horrible story." He went on to describe how, in the course of collaborating with a senior figure in the field, Prof X,  he and his postdoc discovered unambiguous evidence that the samples of Prof X were contaminated, and hence worthless. 6 months of work by Greening and his postdoc were wasted. Prof X was not happy to be told this and was determined to press ahead. This led to a break-up of the collaboration but also attempts to discredit and prevent publication of subsequent work by Greening and his postdoc.

This is exactly the kind of thing that needs to be tackled to improve reproducibility of research. Prof X is not accused of fraud: the initial problem appears to just be down to sloppiness, but that will still lead to misleading findings in the literature. The subsequent bullying of Greening just compounds the problem.

Responses to his tweet were sympathetic and some people suggested he ask advice from Elisabeth Bik, who is a well-known "data sleuth" who identifies and tackles fraudulent or sloppy science[12]. But this is totally unsatisfactory. Not only is Bik far too busy to take on another case, but she is also a private individual who has no formal role in research evaluation and no powers to act. Indeed, a recurring complaint that she and other data sleuths have is that once they reveal unambiguous evidence of errors in papers, it is hard to get anyone to take action. Indeed, my impression is that most publishers regard retraction as appropriate only if misconduct is confirmed by an institutional inquiry - cases of sloppiness and/or "honest error" are seldom acted upon, perhaps because of concerns that if action were taken, this would open the floodgates to numerous criticisms of published work.

We need an agency to whom the problems with Prof X's research could be reported, so they could be evaluated and action taken if the accusations are found credible.

Relatedly, it is concerning that Greening concluded his tweet saying: "I won’t give names or reveal details that implicate the person responsible." So it is clear we have a classic case where a potential whistleblower is scared of repercussions (which could include legal threats), and so nothing will happen and Prof X will continue merrily on his way, polluting the literature with dodgy science and no doubt building up a reputation as a productive scientist on the basis of irreproducible findings.

One route might be to contact the research integrity officer of Prof X's institution, but there are ample precedents of institutions failing to act in such circumstances. Where criticisms are raised about a researcher who is powerful and brings in substantial research income there is stark conflict of interest. Concerns about reputational damage and potential loss of research stars, mean that institutions will kick complaints into the long grass, leaving whistleblowers vulnerable and frustrated.

One cannot, of course, assume all accusations of shoddy research are justified: there are instances where such accusations are used in an attempt to silence researchers who pursue specific lines of work (as in this example[13]). This is precisely why an independent, trusted body is needed to evaluate cases.

B. Action by academic publishers.

Act more promptly to retract or add expression of concern to papers with clear errors.

It is shocking how long published papers can remain in journals after serious flaws that invalidate the conclusions have been unambiguously demonstrated[14].  Publishers appear more focused on the needs of authors than readers, and do not seem to have any sense of responsibility for ensuring that the literature is accurate. They have understandable concerns about legal challenges from authors, but these could perhaps be dealt with by requiring authors to sign a pre-emptive disclaimer asserting the publisher's right to either retract or add an "expression of concern" to a paper if serious errors are found post publication.

C. Action by academic institutions

Training courses for detection of problematic data

An innovative solution would be to encourage a UK university to set up an MSc course on identifying and evaluating bad research. Currently, much of the work on data sleuthing is done by a handful of individuals on a voluntary basis. Methods have been developed for identifying when patterns of data are impossible[15], when biomedical images have been manipulated[16], or when text or images have been recycled[17]. We need to train more people with expertise in these methods - who could develop additional methods in order to be able to plausibly operate the kind of unit proposed under section A.

D. Action by funders

Limit the amount of funding held by a single individual

As noted above, the purpose of gaining grants has become corrupted, so that instead of being a means to do research, grants are seen as symbols of success. This has led to a situation where a few highly powerful people hoard grant money and build up big research groups, but without maintaining quality control. I recommend that funders look carefully at the possibility of limiting the amount of funding (from all sources) that is held by one person. We don't want to stifle good research that needs substantial funding, but the best research is done by groups where the lead researcher is involved in the research, and not just juggling a large number of grants. 

Encourage collaborative studies to incorporate replication and/or converging evidence to tackle big questions

Many of the big questions we are confronted with are unlikely to be solved by a small research group working in isolation. A different funding model might be considered whereby key questions are identified, and groups invited to propose research to tackle them - then, rather than selecting just one for an award, the aim would be to fund groups who either adopt complementary approaches, yielding converging evidence, or who work independently to run the same project, to ensure that results are replicable (or if not, that we are aware of this). A recent Nature paper on biomedical studies[18] illustrates the problems that arise from reliance on individual experiments by small research groups, and proposes a different kind of model, with confirmatory experiments by collaborating groups, to ensure robustness and generalisability.

How could a national committee on research integrity under UKRI impact the reproducibility crisis.

It is difficult to evaluate this idea without knowing the remit and powers of the committee. It certainly would be good to signal that reproducibility issues are being taken seriously at the highest level, and this would help combat the discouragement felt by many early-career researchers who feel there is conflict between doing good science and career progress.

My sense is we know what the problems are - what we lack are formal routes to evaluate and act upon cases where individual researchers or organisations appear to have acted in ways that adversely affect reproducibility. We need mechanisms that would encourage institutions or publishers to take measures to address the counterproductive incentive structure in this area. If the role of a committee were to evaluate potential actions and recommend funding for them, this could help place the UK in the forefront of bringing about change to what has become a dysfunctional system.

(September 2021)