Written Evidence Submitted by the University of Manchester

(RRE0028)

Introduction

The University of Manchester welcomes this inquiry into reproducibility and research integrity. As one of the largest research-intensive Universities in the UK ( £121M in competitive Research & Innovation Grants and Fellowships awarded by UKRI in 2020/21) we recognise the need for research to be conducted and reported in a fully transparent manner. We are institutional members of the UK Reproducibility Network (UKRN) and welcome the recent Research England Development Fund award to support the UKRN’s activities. As an institution we are committed to research, learning, and innovation that brings benefits to society and the environment. Indeed, the goal of creating an open and responsible research environment is one of our five strategic priorities for Research and Discovery. As part of this we are transforming our research environment to support Open Research and a modernised Responsible Research Framework for how we organise, resource, conduct and share our research to meet the highest standards of research conduct and integrity; environmental sustainability; equality, diversity and inclusion; and positive economic and social impact. We are a signatory of DORA, are fully committed to the responsible use of metrics, recently created an Office for Open Research, and released a position statement on Open Research.

Transparency in the research process is key to generating research that will be reproducible. Ideally researchers should view this approach to research as the optimal (and normative) way to conduct research. One of the likely causes of the so-called ‘replication crisis’ in science has resulted from a primarily metrics-driven approach attempting to assess research quality at the level of individual researchers. Despite the intentions of initiatives such as DORA, an individual’s research success still tends to be defined in terms of where their research is published, how many research outputs they generate, and how much grant income they are awarded. None of these measures necessarily correlate with research quality and neither are they dependent on the adoption of open and transparent research practices.

We note that a 2016 task force set up by Jo Johnson examined the closely related issue of open research data (a key component of research transparency). We hope that the current inquiry takes on board the detailed recommendations of that task force. The importance of open data (specifically in the life sciences) and the role it plays in driving innovation was highlighted in the recent ELIXIR report.

The need for skills and infrastructure

Engaging in transparent research practices that will result in scientific discovery that is reproducible is often dependent on a research group having individuals with the necessary computational and data skills. Open research professionals such as data stewards and research software engineers play a critical role in the adoption of open and transparent research practices (arguably a prerequisite for increased research reproducibility and research integrity). These career paths must be formally recognised and supported by institutions, and the individuals on these paths must be funded specifically for this activity. One option is for UKRI councils to require the inclusion of costed activity related to research transparency in funding requests, with a clear and detailed plan for how researchers will make their research transparent (and thus reproducible). This activity must be funded.

During their training as PhD students and as post-docs, researchers must be introduced to issues related to transparency and reproducibility in science and provided with the appropriate skills to ensure the research they conduct does not suffer from the same issues as research of the previous generation. Training must also be made available to more established researchers (i.e., mid- and later-career researchers) who perhaps will need to unlearn sub-optimal research habits which may have led to limited research transparency in their existing programmes of research. In many institutions the pressure to publish in journals with high impact factors is greater than the pressure to publish research that is open, transparent, and reproducible.

Research Integrity

Currently, it is Universities who are responsible for investigating research misconduct (e.g., following the discovery of Questionable Research Practices (QRPs) in a research project that results in a journal publication being retracted). The current reward structure means there is incentive for researchers to engage in QRPs in order to increase their chances of their work being published in a high impact factor journal. Indeed, there is little incentive not to engage in such behaviour. The academic incentive structure should reward good research conduct (and thus disincentivise the use of QRPs). We welcome the creation of CORI within UKRI as an independent body to examine issues around research integrity. We propose that the new UKRI CORI review how best to establish governance and accountability for the sector to ensure that research ethics and integrity standards and procedures across the sector are developed to (i) align with Open Research principles and practices and to remove incentives for QRPs (ii) reward good research conduct (iii) develop and maintain exemplary standards and practice for all aspects of research ethics and integrity.

Incentivising the right things (and incentivise institutions as well as individuals)

Universities should recognise and reward the adoption of reproducible and transparent research practices in hiring, probation, performance reviews, and promotion. This is currently not widespread - we suspect partly because there is less incentive to do this than there is to reward grant income, publications in high impact factor journals, etc. Senior University leaders need to be able to properly understand the extent to which their researchers are engaging in reproducible and transparent practices.

A metric-based approach to assessing research quality often encourages researchers to publish as much as they can, in as many high impact factor journals as they can. It is ultimately an individual researcher’s responsibility to ensure they conduct and report research openly and transparently, but institutions need to incentivise this behaviour. Similarly, institutions need to also be incentivised to ensure that the science their researchers generate is also open and transparent. A positive research culture is key to successful research activity: “research organisations cannot perform at the elite end of research performance (as assessed by the REF) unless they have a positive research culture”.

It is important to recognise that research transparency will have a different meaning for different research disciplines. Domain specific toolkits for research transparency will need to be developed by research communities for those research communities. This activity must be incentivised and funded.

The Role of Publishers in Addressing the Reproducibility Crisis

The use of pre-prints by researchers has an important role to play in improving the reproducibility of research. Pre-prints allow the research community to have an ‘early view’ of research results and encourages peer-to-peer communication about all aspects of the reported research. This can help highlight issues related to research transparency at an early stage of the publishing process.

The Role of UKRI

UKRI has done an effective job in encouraging journal articles themselves to be openly available (via its newly announced open access policy), but has not yet addressed the challenge of ensuring that the research in these outputs is itself open, transparent, and reproducible. The Research Excellence Framework (REF) has a key role to play in ensuring that the outputs submitted by institutions for assessment report their research methods, analysis code, FAIR data (and metadata) in such a way that is as open and transparent as possible. Funders and Government need to adopt incentive models that reward the adoption of open and transparent research practice. Research workflows that are closed and opaque are research workflows that cannot be reproduced.

We would not want to see a situation such that in order to be eligible for UKRI funding, individuals must have had reproducibility training and universities must have certain structures in place. There is a danger that it will degenerate into a “box ticking” exercise and fail to address the central issue. We could end up with researchers engaging in performative research transparency. The best solution to increase research reproducibility and research integrity is for individuals and institutions to be incentivized to change their behaviours. Open, transparent, and reproducible research needs to be funded and rewarded. This is currently not the case. UKRI should play a key role in ensuring appropriate incentivisation (e.g., through the funding councils and through research assessment in the form of REF).

The role of a national committee on research integrity

We welcome the creation of UKRI CORI but recognise that issues around research transparency are not necessarily the same as issues related to research integrity. In terms of researcher behaviour, focus should be on developing norms around research transparency, a process that supports/mandates these, and a culture that values people who work to improve on norms. A national committee should not be about ‘how scientists should behave’ rather about ‘how we should do science, and the process for getting there’. We also need to look at how other countries are addressing the same issue. For example, France recently launched the Second National Plan for Open Science. This builds on their 2018 National Plan for Open Science and is part of a long term strategic vision for how science should be conducted. In terms of having a coordinated national policy, the UK is lagging.

Contributors

This response was generated by the following contributors:

Professor Colette Fagan (Vice President for Research)

Dr Andrew Stewart (Institutional Lead for Open and Reproducible Research)

Professor David Eisner (Division of Cardiovascular Sciences)

Professor Carole Goble (Department of Computer Science, Joint Head of Node ELIXIR-UK))

Professor Christopher Jackson (Department of Earth and Environmental Sciences)

Professor Caroline Jay (Department of Computer Science)

Professor Andrew Trafford (Division of Cardiovascular Sciences)

(September 2021)