Written Evidence Submitted by Dr Charlotte Brand, University of Sheffield
(RRE0026)
I am a Postdoctoral Research Associate at the University of Sheffield in the Department of Psychology. I have previously worked at the University of Exeter, and the University of St Andrews.
This submission reflects my personal experience as an early career researcher, and how I see these problems impacting other researchers who are also on temporary, precarious contracts, seeking grant-funding or permanent research positions.
Research integrity and reproducibility are threatened by two key, intertwined problems in the current research system: 1) there is a lack of transparency throughout the research process and 2) scientific publishing relies on a faulty prestige economy maintained by the current publishing companies. Both of these problems threaten not only the integrity and reproducibility of research outputs themselves, but also the retention and career progression of talented and rigorous early career research scientists within public research institutions.
Problem One: Lack of transparency throughout the research process
The first key problem is the lack of transparency throughout the research process. As an early career researcher I was surprised to discover that I was not required to submit my data and analysis code when publishing in peer-reviewed journals. The current system for checking the reliability of research outputs is the peer review system, which usually operates on the completed manuscript alone. This manuscript is the end product of any number of analytical decisions made by the authors: they can alter their research plan after they have seen the data, and have the freedom to analyse and re-analyse the data indefinitely until they get the result they desire or expect. It doesn’t require any statistical training to understand that if a dataset is analysed in a multitude of ways, apparent ‘results’ may be produced solely by chance. Selectively communicating the analysis that produced a ‘result’, rather than selecting an analysis based on theoretical motivation or best statistical practice, therefore creates a proliferation of untrustworthy and ultimately unreplicable findings [1,2].
The ‘peer review’ system is rightly touted as the best current system for producing independent, reliable scientific work and is why relying on peer-reviewed publications for informing policy is recommended. The current peer review system is, however, buckling for a variety of reasons. One such reason is that peer-reviewing is voluntary and relies on indirect reciprocity; there is no incentive to complete the peer review process rigorously or fairly, or indeed to dedicate the necessary time to this crucial part of publishing research, aside from a sense of moral obligation. Another reason is that individual reviews are not open or transparent but happen behind the ‘closed doors’ of a particular publication, leaving ample room for reviewers’ own biases to affect the interpretation of results. Thirdly, there is currently no way to ensure that reviewers have thoroughly checked every aspect of the research process, namely: (1) that the experimental design and method of data collection were appropriate for the question being asked, (2) that the methods and analysis were appropriately chosen and performed correctly, (3) that the results are valid and reproducible and (4) that the conclusions made can be reliably drawn from the data collected. As it is not the norm for data or analysis scripts to be included with the peer-review process, let alone a pre-registered analysis plan, it cannot be verified that these steps have been taken for any given publication currently standing in the scientific literature. Thus, the trust we place in currently 'peer-reviewed’ scientific literature, and the trust we communicate to the wider public, may often unfortunately be overstated.
Problem Two: Scientific publishing relies on a faulty prestige economy
The second key problem is that the current publishing system relies on a faulty prestige economy that is misdirecting a huge amount of public money. Not only was the current publishing system created when papers were physically printed and distributed to physical libraries, it was also created at a time when far fewer researchers were practicing and publishing research. There are now over 2.5 million papers published every year, with between 20-40,000 scientific journals in circulation. This vast amount of knowledge proliferation requires digital infrastructure and digital distribution which the traditional publishing companies are not adequately providing; the system still rests on an outdated model of paying to read a physical publication. Under the current system, publishing companies host content that is given to them for free by publicly funded researchers, the content is reviewed and edited for free by publicly funded researchers, and access to this content is then sold back to publicly funded universities via subscription fees. UK Universities spend approx £100 million a year on access costs with big publishers like Elsevier, and these costs are increasing despite a lack of any transparency on what the costs are for.
The ‘prestige economy’ refers to the fact that researchers are neither paid to publish, nor to review publications. Instead, researchers acquire prestige for publishing articles in particularly prestigious or highly regarded journals. These journals are deemed to be of ‘wide interest’, such as renowned titles ‘Nature’ or ‘Science’ that may be familiar outside of academic circles. Prestigious journals typically have a high “impact factor” and a high rejection rate for submitted articles, (indicating that the journals are selective and elite). However, these “high impact” journals do not produce publications that are of higher quality, nor do they provide a more rigorous review process [3,4]. What these journals appear to prioritise is the novelty or newsworthiness of findings, rather than the rigour of the underlying research. Indeed, studies repeatedly demonstrate that publications in such journals are more likely to be later retracted due to faulty analysis or even fraud [5,6]. When researchers are competing with each other to produce the most novel and newsworthy findings, incentives to cut corners are created. With no malign or malicious intent, researchers may unconsciously exaggerate their findings, or keep re-analysing data until they get the desired outcome they want to communicate [2]. Publishing in a prestigious journal, then, is less an index of research quality than an index of potential un-replicability. In this way, the “impact factor” can be viewed as a successful marketing strategy on behalf of the publishing companies; it has created competition amongst researchers for space in journals that communicate scientific novelty, rather than scientific value.
Because universities and funding bodies hire and award researchers based on the number and impact of their publication record, they are creating a selection pressure for researchers who generate newsworthy, novel, and ultimately unreplicable findings [2]. It means that scientists are having to compete with each other not to produce the most rigorous, careful and informative work, but to produce as many high impact publications as possible in the shortest amount of time. Furthermore, the vast majority of research is done by early career researchers whose continued employment is conditional on their publication record. In other words, the pressure to produce unreplicable findings is strongest on the people who do the bulk of the research. Scientists who do their research with the most care and rigour, and who most strenuously resist ‘questionable research practices’ (practices such as repetitively re-analysing data mentioned above) are those with, currently, the slimmest job prospects, because they necessarily produce fewer, ‘lower impact’ papers.
All of the above leads to three fundamental problems that threaten the integrity and reliability of research conducted in the UK and worldwide. 1) We cannot trust the validity of a huge amount of the scientific literature 2) We cannot retain the most dedicated and rigorous research scientists 3) A huge amount of public money is being wasted as it is funnelled directly into publisher’s profit margins rather than being used to maintain integrity in the publishing system.
Solution One: mandate transparency at all levels
Funding bodies, research institutions, and publishing companies must mandate transparency throughout the research process. This is thankfully beginning to happen, thanks to more and more institutions signing the DORA declaration, and funding bodies such as the Wellcome Trust and UKRI mandating open publishing of work funded by them. However, it needs to be ubiquitous, so that it is a norm rather than a choice, and audited. One solution that has been proposed and embraced by some fields (mostly psychology) is ‘pre-registration.’ Preregistration involves publicly registering all data collection protocols and analysis plans, particularly hypotheses and predictions, before the data are collected or seen, as is already default for clinical trials. This step could be mandatory for all research in which it is appropriate - specifically hypothesis-testing experiments - and could be mandated by research institutions, publishers, and research funders. It is of course natural for research plans to change as a project progresses; be that changing questions, adjusting collection protocols, or even altering experimental design. However, preregistration does not preclude change (a common misconception), but instead requires honest, chronological documenting of the changes along with their justifications. This documentation not only aids peer-review, but is also extremely valuable for other researchers who are undertaking similar projects.
Solution Two: require transparency from for-profit publishing companies
The publishing companies who provide the current infrastructure for scientific publishing are, as it stands, not creating any added value to the research outputs other than organising peer review and hosting the publications. Currently, a researcher may publish their work before peer review, for free, on a preprint server, and allow the scientific community to openly critique their work for free (e.g. via social media). In many ways this would be of greater benefit to the researcher, as their work will reach a wider audience and, arguably, be more thoroughly and heavily reviewed.
Under the traditional system, most peer-reviewed work can only be read by researchers who work for an institution that holds a subscription to that journal. Alternatively, a researcher may choose to pay an Article Processing Charge (APC), created by the publishing companies in response to the Open Access movement, to ensure their article is openly available to all, irrespective of whether they hold a subscription to the journal. This “pay twice” system is unsustainable, unjustifiable and demonstrably unfair as it means publicly funded universities are paying the publishing companies twice; once for their access fees, and again to publish open access. This problem has thankfully been recognised in recent UKRI policy to only pay APCs in the case of ‘transformative agreements.’
The publishing companies should be regulated to ensure that 1) the costs for the subscription are transparently laid out and justified 2) all article processing charges are transparently laid out and justified 3) universities can choose to either pay subscription fees for access to the closed content of journals whilst submitting their open access articles for free, or pay APCs when publishing their articles, which allows them access to the closed content of those journals. Thankfully, due to the negotiation of university libraries, an increasing number of deals now allow this. However, university library staff need the explicit support and endorsement of such plans from academics to get into a negotiating position to begin with, rather than being trapped in the status quo of extortionate fees.
Solution Three: add value to the publishing system by paying expert reviewers
As it stands, the publishing companies are not contributing value to the final publication that is produced by researchers. To add value, publishing companies could provide services that are currently lacking: 1) hire teams of dedicated, trained, specialised postdoctoral peer reviewers to complete rigorous, trustworthy reviews 2) hire specialised statistical editors to check the reproducibility of data analysis 3) hire data scientists and engineers to manage and maintain the archiving of statistical analyses, data, and articles.
The benefits of ensuring the above solutions are threefold; 1) researchers from both academia and industry can rely on the scientific record and make use of research outputs with assurance that the research record is reliable and trustworthy 2) universities can save public money by only paying justified, proportionate costs to publishing companies. Saved costs can be put back into a university publishing infrastructure, or to fund schemes that help attract the most talented and rigorous researchers from diverse backgrounds 3) talented and rigorous researchers will feel secure and valued in a research system that prioritises rigorous reliable science, and that offers them more secure career opportunities within scientific research.
A national committee on research integrity under UKRI could positively impact the reproducibility crisis if they:
• Ensure that publishing companies are held to account. Namely, that they 1) require data and analysis scripts to be submitted alongside the manuscript for peer review, 2) ensure reviewers are reading and reproducing the analyses, 3) monitor and audit their peer review process to identify and eradicate instances of clear bias and malpractice
• Ensure that publishing companies transparently communicate and justify their APC and subscription costs, and evidence how these costs contribute to maintaining reproducibility and integrity of their outputs
• Encourage publishing companies to rebuild their journals’ reputations around quality and rigour instead of novelty and newsworthiness, for example by embracing the innovative Top Factor [7].
• Reccomend funding for initiatives to develop and design alternative publishing systems, including peer-review reform and other innovative initiatives to increase integrity
• Recommend funding for initiatives that educate academics on the financial and reputational harm of publishing in the current system, and what alternatives there are
The impact on Early Career Researchers
The ‘natural selection of bad science’ [2] that has been happening for many decades (arguably since the introduction of the impact factor) has created a research culture that is, at best disappointing, and at worst toxic, for early career researchers. Many researchers embark on their PhD training with a sense of awe and respect for the scientific process, but this is all too often eroded and replaced with disillusionment. Every PhD student quickly learns that a PhD will not guarantee you a job as a research scientist, and that you are unlikely to secure a permanent research position for at least several more years, if you do get that lucky. The competition is too high, and the jobs are too few. But putting job security to one side, the pressure to publish as many novel findings as possible, together with the knowledge that the scientific literature you are contributing to may not actually be trustworthy, is itself psychologically crushing, and makes academia not only unappealing but often repellent for those early-on in their career. What’s more, the pressure to publish in these unreproducible ways can come directly from your own mentors, with accounts of bullying and misconduct in scientific labs all too common. Sentiments like those shared on social media just this week [8], in which early career researchers are psychologically worn-down over time until they succumb to the pressure of falsifying their results, for the sake of novel and noteworthy publications, are unfortunately not as rare as we would like to believe.
I personally have witnessed early career researchers being bullied out of academia due to a desperate commitment to the current publishing system, but I have also personally been extremely lucky with my own mentors. If it wasn’t for consistently supportive mentors, who not only speak-out against the current system but embrace and implement changes to it, I would be joining many others in leaving the university sector. Neither of my parents, nor my family members have been to university, yet as an undergraduate I fell in love with science to such an extent that I pursued a masters degree, a PhD and now postdoctoral research. I couldn’t believe such a job as ‘lecturer’ was possible and aspired to it since my own undergraduate lectures. I am only eager to continue on this career path due to the positive changes that I have seen being made: institutions signing the DORA agreement, institutions making Open Science pledges and signing-up to organisations such as UKRN, and funding agencies enforcing Open Science policies. If funders and institutions can take the next step on this positive path and hold the for-profit publishing companies to account, I believe this would pave the way for a flurry of innovations that could improve research culture for scientists of all stages.
REFERENCES
[1] Open Science Collaboration. (2012). An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science, 7(6), 657-660.
[2] Smaldino, P. E., & McElreath, R. (2016). The natural selection of bad science. Royal Society open science, 3(9), 160384.
[3] Seglen PO (1997). Why the impact factor of journals should not be used for evaluating research. BMJ. 314 (7079): 498–502.
[4] "EASE Statement on Inappropriate Use of Impact Factors". European Association of Science Editors. November 2007. Retrieved 13 April 2013.
[5] Brembs, B., Button, K., & Munafò, M. (2013). Deep impact: unintended consequences of journal rank. Frontiers in human Neuroscience, 7, 291.
[6] Brembs, B. (2018). Prestigious science journals struggle to reach even average reliability. Frontiers in human neuroscience, 12, 37.
[7] Mayo-Wilson, E., Grant, S., Supplee, L., Kianersi, S., Amin, A., DeHaven, A., & Mellor, D. (2021). Evaluating implementation of the Transparency and Openness Promotion (TOP) guidelines: the TRUST process for rating journal policies, procedures, and practices. Research integrity and peer review, 6(1), 1-11.
[8] “I Am an Academic Fraud” (2021) posted anonymously on Reddit 28/09/21, accessed via https://twitter.com/Jake_A_Lacey/status/1442666121618882561
(September 2021)