Written Evidence Submitted by Elsevier

(RRE0049)

Introduction

Elsevier is a leader in global information analytics, helping researchers and healthcare professionals advance science and improve health outcomes for the benefit of society. We do this by facilitating insights and critical decision-making for customers across the global research and health ecosystems.

We welcome the opportunity to respond to the Science and Technology Select Committee’s Inquiry into reproducibility and research integrity. In everything we publish, we uphold the highest standards of quality and integrity. In particular, we play an active role enhancing and disseminating reproducible research, and we share our insights below for consideration.

Executive Summary

1. While reproducibility is crucial to enable knowledge to progress, we note that there are specific problems and challenges in ensuring that all research studies can be reproduced.[1] The problems in academia related to reproducibility are varied; at times localised to individual research disciplines, but also systemic in the way that research is carried out, for instance due to variations in methods and processes, or in the natural deviations in the subjects being studied. While research practices do need improvement, we do not suggest that the problems in reproducibility are to such a scale that they have irrevocably impacted scientific research, or should necessarily be characterised as a crisis.[2] Indeed, the scientific research process as a whole is highly successful, reliable and rigorous – the success of the development of the Covid-19 vaccinations being a clear example of how scientific research works to the benefit of society. Many challenges in reproducibility are endemic in the research process, and therefore need holistic solutions alongside cultural change.
2. Publishers, with a range of other stakeholders in the research community, have a clear role to play in improving reproducibility practices. Publishers can specifically focus on enhancing transparency and rigour in the research process, for instance by funding, organising and manging the governance of peer review; empowering researchers to deploy transparent processes and methods; and incentivising good practices, such as data sharing.
3. Reproducibility challenges can inherently occur throughout the research process, and good practice must therefore be encouraged at every stage of a research project, rather than only at the point at which scientific results are ready to be published and disseminated. We therefore recommend that all stakeholders in the research ecosystem more closely align their objectives, policies and incentives to improve reproducibility practices across the research process, and influence research culture to that end. We also welcome a UKRI national committee, which can work to meet these goals.

Response

1. The challenges in reproducible research: breadth and issues in academia

Publishers and stakeholders across the research community fully recognise that we move research forward by reproducing and building on past research. Reproducibility is therefore crucial to enable knowledge to progress, and to ensuring productive and efficient practices in the research process.[3] We at Elsevier recognise that reproducibility has specific challenges in that not all research studies can be repeated,[4] and we acknowledge the potential consequences: should inaccurate research findings become part of the literature, more studies will attempt to build on tenuous foundations. If these problems persist in replicating research, this could, for instance, slow development in medical treatments, misguide government policies, and ultimately impact wider society.

However, we do not suggest that that the problems in reproducibility are to such a scale that they have irrevocably impacted scientific research, or should necessarily be characterised as a crisis.[5] Notwithstanding the challenges, scientific research continues to yield measurable improvements to daily human life – the development of the COVID-19 vaccinations being a case in point. Moreover, the challenges in academia related to reproducing research are diverse and often down to the character of individual disciplines, or the production of research, which is naturally subject to variations and human error. Challenges are often inherent in research processes and therefore require holistic solutions across research culture (explored further below). The challenges include:

•                                                                                                                                                                                                                     Reproducibility practices and the extent to which they are implemented can vary across different disciplines. For certain disciplines, researchers absolutely support the need for reproducible research and good practices such as data sharing, but for valid reasons they are confronted with barriers, relating to privacy, consents from patients or other research subjects, or preventing misinterpretation of data.[6] Sharing research data is particularly challenging in medical fields because of important issues around privacy, plus the costs and resources to responsibly de-identify data. Psychological studies can be more difficult to reproduce due to natural variations in human behaviour and low statistical power. Finally, the combination of reproducibility and AI used in research has its own challenges[7] and requirements. Not only do researchers need to share code,[8] software and computing details to reproduce any computational experiment, for (machine) learning systems, they also need to provide information on any and all (training) data used, including  its provenance and pre-processing steps; a description of methods used to select hyper-parameter configurations; and detailed specifications of all hyper-parameters used to generate results.
•                                                                                                                                                                                                                     Tied to the above, researchers can also face a range of challenges when working to create reproducible research, which have nothing to do with unethical or faulty processes on their part. In our experience, researchers are rarely falsifying their research, but mistakes or wrong conclusions may be a natural yet all too common consequence of conducting research. A lack of reproducibility could be down to an antibody that was not as stable as a researcher would like, for instance, or contaminated samples, or slight variations in methods – all contributing to differing results. Further, research itself and the questions that researchers are seeking to address are increasingly complex and challenging, while potentially being expensive to design and implement, which again has knock-on effects for reproducibility. Some of these challenges can be very much improved with education, and emphasis on transparent methodology. But solutions must be targeted where possible, working with the grain of individual research communities and their methods and culture. Importantly, we should work to avoid stigmatising researchers if certain studies cannot be easily reproduced.
•                                                                                                                                                                                                                     Multiple stakeholder roles are required to provide and fund incentives, set standards and change behaviours on reproducibility of research. Stakeholders include funders, institutions, publishers and researchers. These incentives also need to take place across the research process, from idea to publication. We therefore welcome moves to bring stakeholders together and align on solutions via a national committee. 
•                                                                                                                                                                                                                     Finally, there can be a lack of incentives, education, and funding to consistently encourage researchers across the research process (i.e. from idea to publication) to adopt reproducibility practices. This can contribute to researcher behaviours that don’t prioritise reproducibility; for instance, reluctance from researchers to share data openly, or to publish research that demonstrates negative or null results. This reluctance is reinforced through reward systems focused on narrow measures of success, leading to publication bias (when the results of a study too easily determine whether the research is published) and other adverse consequences.

In the next section, we explore the range of solutions that we have embarked upon to address some of the above challenges.

2. The role of publishers in addressing reproducibility

Publishers have a clear role in maintaining the integrity of the scholarly record, and thereby ensuring the quality, reliability, and reproducibility of research. Primarily, we ensure scientific rigour in our published content through our editorial and peer review processes. We also support the long-term preservation of trusted and reliable research, making investments to ensure that the scientific record is up to date, and enabling publishers to implement editorial decisions to correct or retract inaccurate or unethical outputs. In turn, published research enables researchers and practitioners to deliver impacts for society.

Reproducibility forms an important pillar of our commitments to research integrity, and is underpinned by our efforts to encourage openness and transparency in the research process: reproducibility can only be effective if researchers are fully transparent in how studies and experiments were carried out, the methods and protocols involved and in fully describing the results.

Publishers therefore invest in a range of activities which are designed to encourage reproducible research. We use the peer review process to interrogate authors’ use of methods, results and statistical analysis; we support open science practices across the research workflow to encourage transparency and accessibility of all kinds of research outputs; our journals support the setting of standards around scientific rigour and empower researchers to share methods and processes; and we can be pivotal in providing incentives and creating platforms to enhance reproducibility practices. Examples of work that we at Elsevier have undertaken are below.

• We use the peer review process to foster innovations in journals and new peer review models, which supports evaluation to enhance reproducibility of research. Publishers’ management of the peer review process includes providing tools and information to editors to find the most relevant reviewers, as well as diversify the background of the peer reviewer pool: expertise and diversity in the peer review process ensures reducing bias, which in turn enhances reproducibility. Editors also have a role in the peer review process to examine, interrogate and critique researchers’ analysis, methods, results, conclusions and statistical analysis, as well as review any research data accompanying the original submission. Each of these components are integral to ensuring reproducible research, and are therefore encouraged via the peer review process. Examples of specific models in peer review are below:

-          We continue to experiment in peer review, for example via Results Masked Review (RMR). The RMR model aims for work to be judged on the merits of the research question and methodology, not the findings. RMR articles are sent for review without the results, discussion or conclusion (although data has already been collected) and reviewers are asked to evaluate the article on the hypothesis and the methodology only.

-          Registered Reports also stem from innovations in the peer review process. As part of the model, authors submit their protocols before experiments are conducted. Importantly, this means that authors have committed to the protocol they have submitted and how they conduct their research. The journal then accepts the paper in principle, based on whether editors believe the protocol has merit.

• Registered Reports support reproducibility as the journal has committed to publish research regardless of the results. This prevents publication bias and enhances transparency, thereby ensuring that results aren’t skewed to facilitate publication and can subsequently be reproduced.
• While Registered Reports are increasing and are successful in eliminating some key challenges in reproducibility, take-up from researchers has been slow. Reasons for this include the dramatic change that Registered Reports represent to the publication process. Authors are understandably concerned about being tied in to publishing a paper that may bear results they hadn’t predicted, which could in their view hamper innovation, or their careers.

-          Finally, we collaborate across the industry and develop a range of pilots within peer review to promote transparency and accountability, as a clear factor in supporting integrity and, by extension, reproducible research.

• We invest in a range of systems to ensure the integrity of the research findings that we publish. Examples include ongoing investments and collaboration in working groups across the industry to detect fake articles and tackle image duplication and inappropriate image manipulation. Clearly, if not checked, these issues could have a highly detrimental effect on reproducibility, and we continue to explore systematic processes to identify potential problem areas.
•                                                                                                                                                                                                                     Research data are the foundation on which scientific and medical knowledge are built. Access to data that underpins published findings ensures that the research can be successfully reproduced. As a key pillar of Open Science, and to uphold research integrity, we promote and encourage Open research data sharing practices and incentives for authors. These include:

-                                                                                                                                                                                               Providing products and platforms to incentivise researchers to share their research data in a way that is structured, but easy for researchers to deploy. These include Mendeley Data, a secure cloud-based repository for researchers to store, share, access and cite their research data. Digital Commons increases visibility of the full spectrum of research, across research data, articles and other outputs. And Elsevier’s Entellect enables seamless access to reusable, integrated Life Sciences data that adheres to FAIR principles.

-                                                                                                                                                                                               We promote FAIR data principles to ensure that data is findable, accessible, interoperable and reusable. We are a founding member of Force11, which developed the FAIR principles.

-                                                                                                                                                                                               We have also aligned and clarified our journal policies regarding open research data to ensure that researchers understand what we expect of them in terms of sharing research data.

-                                                                                                                                                                                               To help make all scientific data more transparent, a number of our journals encourage authors to state the availability of their data – functionality that will be rolled out across the majority of our journals in the next year. With the data availability statement, authors can be transparent about the data they used in an article and make a statement about its availability together with their published article.

-                                                                                                                                                                                               We have integrated data sharing incentives to our submission workflows, thereby using our journals to influence researcher behaviours and incentivise them towards data sharing. For example, we encourage data sharing at the point that a researcher submits their paper to our journals. We have found that by simply making it easy for authors to data share, and reminding them early on in the publication process in our submission workflows, this has doubled the amount of data sharing to support reproducible research.

• We invest in journals as a way of incentivising authors to data share and publish different kinds of research outputs. By ensuring that data and research outputs are shared openly and transparently, this better enables research to be reproduced. Current pilots include:

-          Research Elements journals are dedicated to publishing research elements articles, making the process of scientific research more transparent by publicising the output that has come about as a result of the research lifecycle, including data (see next point below), methods, protocols, software, hardware and more. Journals such as Software X are designed to help serve Research Software Engineers as well as other scientists produce software and code, and participating journals have published hundreds of Original Software Publications.

-          Data in Brief provides a way for researchers to easily share and reuse each other's datasets by publishing data articles that thoroughly describe data, make it easier to find, and enhance collaboration. These objectives contribute towards and underscore reproducibility.

-          We have launched journals that publish negative results, encouraging authors to avoid publication bias and incentivise rigorous scientific practice. Unfortunately, even with strong editorial support and backing, we have had low take-up of these journals, suggesting that researchers have a low appetite to publish these kinds of research outputs. Our feedback from researchers suggests that they don’t feel it was worthwhile to write up negative results, or don’t want to be associated publicly with them. In the same vein, we encourage replication studies to be submitted to existing journals. We also piloted via a group of journals with calls for papers to invite replication studies. Again, we have experienced somewhat low uptake from researchers to date, but this is something we continue to market and build upon.

• We use our journals to incentivise researchers to be transparent in their methodology, which in turn supports reproducibility as others can follow a clear and reproducible method. In this vein, we launched STAR Methods via Cell Press (using a well-known and reputable journal to drive uptake). STAR Methods outlines the features of a robust, reproducible method: structured, transparent, accessible reporting. STAR Methods has proved to be highly successful, having received positive feedback from editors, reviewers and researchers, and now rolled out across 1500 journals.
• We develop initiatives in research assessment and evaluation, for example, by creating the International Centre for the Study of Research and signing DORA. Through the ICSR Lab, we enable no-cost access to the data that power Elsevier’s products and solutions in a cloud-based computational environment. ICSR Lab has reproducibility built-in owing to its data retention policies and change-tracked coding notebooks.
• Finally, we see supporting and enabling diversity in research as crucial towards ensuring reproducibility. A lack of diversity in research processes has inevitable impacts for the reproducibility of research findings and their subsequent accuracy. For example, if clinical trials are only carried out on men, there is strong likelihood for divergence if they are repeated with women. This also has clear implications for benefits to society, if products generated from research do not have diversity and gender taken into account. Elsevier has undertaken a range of activities to support inclusion and diversity in science, including a Resource Center to provide free access to research, data, and tools related to gender; supporting gender balance across our editorial boards and the research community; and increasing awareness of these issues through our range of reports.

As the above list of our activities shows, over the years Elsevier has been experimenting with a range of solutions to meet reproducibility challenges. However, our pilots are subject to differing results and degrees of success. Some negative results journals, for instance, have seen low take-up from researchers, while our data journals have been highly successful. Equally, while publishing replication studies supports reproducibility, this is not suitable to deploy across every type of journal in our portfolio, and it is not always taken up by researchers. Further, while we provide a range of incentives to data share, researcher responses are not always in line with the related policy objectives. As per the challenges in reproducibility listed out above, researchers simply aren’t always able to share data, or are cautious about doing so, often for valid reasons relating to privacy, reasonable concerns about misinterpretations of data, or the logistics and costs involved. These challenges indicate that we need a range of targeted solutions to meet the needs of the diverse communities that our journals support, including the range of concerns coming directly from researchers regarding reproducibility practices.

We will nevertheless continue our work to promote reproducibility. By way of examples, we have rolled out several key elements of our STAR Methods to 1500 journals. We are currently implementing changes to our systems and expect to roll out functionalities relating to data sharing, for instance applying data availability statements and enabling FAIR data sets, across the majority of our journals by March 2022. We continue to test and learn from the results of our journal/ article pilots, while promoting and scaling our ongoing innovative projects like Registered Reports and our Research Elements journals. We have and will continue to take steps to educate researchers, particularly in relation to the above issues reported by researchers in data sharing, using resources such as the ANDS guidelines.

Publishers also have responsibilities elsewhere in the research process to ensure good practices, which in turn support reproducibility. For example, retractions can and do happen – and while these make up a very small proportion of published output, publishers will continue work to prevent this from taking place by educating authors on ethical issues, using detection tools where available and having clear policies. We will also work to ensure that the peer review process is as robust as possible. For instance, where feasible, we can support journals with a statistical sub-editor or designated board member. To further scale this, we are exploring solutions in software and AI to assist peer reviewers’ vital work.

Publishers are nonetheless at times implicated in the noted ‘publish or perish’ mentality, which can give rise to problems in reproducible research. Given the steps that publishers have taken to support holistic research evaluation practices, publishers work to ensure that authors are not pressured to publish reams of work in any single journal. We actively try to educate researchers about these issues, and we work with forums such as COPE to implement standards and best practice to prevent, for instance, salami publications and duplicate submissions. We will continue work with stakeholders in the research ecosystem to explore the drivers and find further solutions to this common pressure amongst researchers.

3.                                                                                                                                                                                                                  What broader policies or schemes might have a positive impact on academia’s approach to reproducible research?

Publishers are but one stakeholder in the research ecosystem needed to promote reproducibility practices. Furthermore, journals are by their nature ‘late’ in the research process, meaning that they are naturally limited in encouraging reproducibility at critical earlier junctures. In our experience, researchers may well have been working on a research project for a long period of time ahead of publication, with many research papers taking years to get to the publication stage. A lack of awareness regarding reproducibility practices at the start of the research process clearly causes huge challenges for repeating studies at a later date. From a practical perspective for instance, it is problematic to ask researchers at the end of their project to share data if at no point beforehand they have been practiced in the art of structuring, organising and labelling their data throughout the research process.

Researchers therefore need education, rewards and incentives to embark on practices that encourage reproducibility from the outset of their research projects. A range of stakeholders, working at different stages of the research process, are subsequently required to incentivise and promote reproducibility practices, such as data sharing. Stakeholders, including funders and institutions, will be operating at these earlier stages of research development and will have important opportunities to influence, appropriately fund and incentivise researchers on reproducibility. Moreover, they could help to raise awareness of the various solutions that publishers offer, in order to reinforce to researchers the importance of reproducibility practices ahead of their submitting papers to journals.

It is therefore vital that stakeholders across the research community align on objectives relating to reproducibility practices. Collaborations across stakeholders should take place in working to build a positive research culture that rewards and integrates reproducibility practices. By way of illustration, the manifesto for reproducible science[9] is a helpful example of how different stakeholders, encompassing regulators, journals, funders and institutions, can work holistically and pragmatically across the research process, to improve reliability and integrity of science. A clear goal in this collaboration would be to ensure that reproducibility practices are naturally integrated into the research process, from inception through to dissemination.

We suggest that we collaborate with stakeholders in the research community to workshop new ideas on reproducibility. This could include the development of incentives, or even mandates, for authors to share data; to agree standards; or to encourage researchers to publish negative/ null results. Workshops or studies could be carried out to examine fundamental questions, including: What would be gained if research was fully reproducible?  What changes regarding research and publishing incentives and infrastructure would be required to make this possible, and which of these has the greatest impact? How do we resolve ongoing researcher concerns around practices such as data sharing? This would be preceded by agreeing on a definition of the concept of reproducibility itself (which is different for different domains, subsequently causing more confusion as to best practice).

Finally, Government support will be helpful to coordinate across stakeholders to drive workshops and action on reproducibility, as well as fund technologies and projects to incentivise reproducibility. However, a key risk we should avoid is that Government over-simplifies the issue or deters researchers further from reproducibility practices (for instance, box ticking exercises which introduce more administrative burdens for researchers).

4. How might a UKRI national committee impact on reproducibility?

We are supportive of a national committee, which could work to implement many of the goals we have outlined above. To be as effective as possible, we would encourage the committee to be cross-stakeholder, to have clear objectives and align on definitions and parameters for reproducibility itself. The committee should motivate and inspire improvements in reproducibility, rather than focus on sanctions or introduce new hurdles that may be cumbersome for researchers. Further, the committee can build on previous partnerships, initiatives and individual stakeholder activities to ensure alignment and better awareness of these activities.

(September 2021)