Written Evidence Submitted by:
John Drury, University of Sussex
Linda Bauld, Edinburgh University
Marcus Munafò, University of Bristol
Jamie Brown, University College London
Zoltan Dienes, University of Sussex
Susan Michie, University College London
Elizabeth Stokoe, Loughborough University
Anne Templeton, Edinburgh University
Robert West, University College London
(RRE0003)
This submission addresses:
We are a group of researchers seeking to help promote reproducible research in our disciplines (psychology, public health) and beyond. Some of us have been involved in research and in an advisory role in relation to the Covid-19 pandemic and are particularly interested in the role of transparency and reproducibility in fostering constructive debate in relation to ongoing research on the pandemic. This submission grew from a response to a specific question on pre-registration and what our institutions are doing (or could do) to support it as a practice. We summarize the different types of support identified and then provide the verbatim responses for each institution/ organization involved.
One area where support is provided is education and training, supplemented by guidance materials where appropriate. For example, the University of Sussex School of Psychology runs seminars for faculty and research students on Registered Reports as part of our Research Methods strategic focus area, which also includes useful published resources: https://www.sussex.ac.uk/schools/psychology/research/environment/psychological-methods
The UCL Tobacco and Alcohol Research Group (UTARG) has a research handbook that outlines their commitment to transparency, routine pre-registration of protocols and open science. At the University of Edinburgh, a pre-registration guidance document is being created for the College of Arts, Humanities and Social Sciences. UTARG also has a shared repository of publications all linked to pre-registration protocols.
However, education and training must be supported by appropriate incentives to drive and embed cultural change around these practices. UCL has an Office for Open Science and Scholarship, with which research groups can engage and make researchers aware that open science practice is included under promotion criteria. At the University of Sussex, mechanisms such as targets (e.g. a particular number of pre-registrations a year) have been discussed, but the School of Psychology has decided not to have a prescriptive policy. One research centre at Sussex has a policy of pre-registering all research, and some faculty have all their research staff pre-register their research protocols routinely (but there are no sanctions for not doing so).
Linked to the issue of incentives is the question of how other stakeholders in the research ecosystem can contribute. Some research funders, including Cancer Research UK, have linked specific research funding opportunities to an expectation that researchers pre-register their study protocols. Cancer Research UK is also currently considering extending a pilot whereby grant review and journal review of a Registered Report is combined into a single integrated process, intended to speed eventual publication of study results, reduce publication bias, improve quality, and reduce overall reviewer burden.
Other funders are also promoting Registered Reports. For example, in a recent funding call, Templeton required that all outputs should be registered reports: https://www.templetonworldcharity.org/our-priorities/accelerating-research-consciousness Other examples of funders and journals collaborating to promote Registered Reports are described in a 2019 editorial in Nature.
Ultimately, ensuring these transparent research practices become normative will require coordination. The UK Reproducibility Network (UKRN) is a national peer-led consortium that aims to support the factors that contribute to robust open science research, promoting training activities, and disseminating best practice. This includes some of the mechanisms mentioned earlier. Nineteen institutions across the UK are currently formal members of UKRN, which means they have established an Academic Lead for Research Improvement, within each university’s senior management, as part of their commitment. UKRN’s open research plan includes training in various open research practices: pre-registration and the use of Registered Reports; preprints; open access; data sharing; and open code and software. These practices are incentivised via an annual open research prize, their recognition in promotion criteria, and their inclusion in job descriptions. Institutional repositories, such as the University of Bristol’s data.bris Research Data Repository, provides the infrastructure to enable the sharing of data, code and materials. Other platforms, such as the Open Science Framework, allow study protocols (and other materials) to be posted. Individual research groups (see example) can then make individual elements of the research process available for scrutiny and re-use. UKRN is also developing training - modelled around the UKRN open research primer series - to provide researchers across disciplines with the necessary skills to engage in open research practices.
Appendix: Verbatim written responses
Sussex (Zoltan Dienes)
Registered Reports are even stricter than pre-registration, and are making inroads into scientific practice. Registered Reports have expanded from Cortex in 2013 to 294 journals now (https://www.cos.io/initiatives/registered-reports). I mention the studies showing Registered Reports reduce bias in the conclusion of my chapter here: https://psyarxiv.com/yhp2a
UCL (Jamie Brown on behalf of Olga Perski, Dimitra Kale, Robert West & Susan Michie)
The UCL Tobacco and Alcohol Research Group (UTARG) has a research handbook that outlines our commitment to transparency, routine pre-registration of protocols and open science. UCL values open science and has an office for Office for Open Science and Scholarship, with which we engage and make UTARG researchers aware that open science practice is included under promotion criteria at UCL. At a group-level, we have fostered an open and transparent culture and senior staff aim to lead by example. As recent examples of recent good practice, here is a list of 2020-21 publications with links to pre-registered protocols and analysis plans: https://liveuclac-my.sharepoint.com/:x:/g/personal/rmjdjb0_ucl_ac_uk/EXAs-CDf7mxCshXeiz5a85QByKKkeq_A-KSgj48FfE__2w?e=enEiLg
Also, we have conducted a living evidence review on the association of smoking status with COVID-19 infection, hospitalisation and mortality, updated 11 times to date and accessed ~52,000 times. Each update is shared directly with Public Health England's ‘tobacco and smoking cell’ and updated on an open science platform (https://www.qeios.com/read/UJR2AW.13), with v7 published in a high-quality peer-reviewed journal (https://onlinelibrary.wiley.com/doi/10.1111/add.15276). The code (https://zenodo.org/record/4572895#.YLpDmvlKg2w) and data (https://docs.google.com/spreadsheets/d/1jelIGxrm99QpDKwSoyMUliDJQWtzk4bHsNGDaIjxTaU/edit#gid=1431343014) underpinning the living review are openly available.
Bristol (Marcus Munafo)
The University of Bristol has established an Academic Lead for Research Improvement, as part of our commitment to the UK Reproducibility Network (UKRN; www.ukrn.org). Eighteen other institutions across the UK are also formal members of UKRN, and have created similar roles. As part of this, we are developing an open research plan, which includes training in various open research practices - pre-registration and the use of Registered Reports; preprints; open access; data sharing; and open code and software. These practices are incentivised via an annual open research prize, their recognition in promotion criteria, and their inclusion in job descriptions. Institutional repositories, such as data.bris Research Data Repository, provides the infrastructure to enable the sharing of data, code and materials. Other platforms, such as the Open Science Framework, allow study protocols (and other materials) to be posted. Individual research groups (see example) can then make individual elements of the research process available for scrutiny and re-use. We are also developing training - modelled around the UKRN open research primer series - to provide researchers across disciplines with the necessary skills to engage in open research practices.
Cancer Research UK (Linda Bauld)
Funders have on occasion partnered with academic journals to encourage researchers to publish registered reports as part of the process of securing research funding. Cancer Research UK did this via their Tobacco Advisory Group (TAG - a funding committee that I previously chaired) with the journal Nicotine and Tobacco Research. Marcus Munafo (see above) is the senior editor of this journal and he wrote about the scheme in a journal editorial. TAG recently merged with the charity’s Prevention and Population Research Committee so the scheme is no longer in operation, but pre-registration is still encouraged.
(August 2021)