Written Evidence Submitted by the Royal Society
(RRE0087)
The Royal Society is the national academy of science for the United Kingdom. Its Fellows include many of the world’s most distinguished scientists working across a broad range of disciplines in academia, industry, charities and the public sector. The Society draws on the expertise of the Fellowship to provide independent and authoritative scientific advice to UK, European and international decision makers.
1. The Society welcomes the opportunity to respond to the Committee’s inquiry into reproducibility and research integrity. The purpose of the Society is to recognise, promote and support excellence in science and encourage the development and use of science for the benefit of humanity. Good science relies on scientists generating and sharing their findings with openness and honesty, which ensures the use of science for social benefit and a culture that fosters research integrity. While definite challenges exist in this area, the Society is concerned by the use of the term ‘reproducibility crisis’ throughout this inquiry as it presupposes the scale of the problem, could narrow the range of evidence submitted and hinder the Committee’s investigation.
2. The Society’s roles include advocate, funder and publisher. In all of these we seek to support research integrity. The Society has led efforts to improve research culture1, has published its own research integrity statement2, and has led work on the role that Open Science can play in deterring misconduct3.
The breadth of the ‘reproducibility crisis’
3. The term, ‘reproducibility crisis’ used in this inquiry presupposes the scale of the problem and can be used by those acting in bad faith to undermine trust in science. Robust investigation, including this inquiry, is vital to understand the breadth of the problem, and what it reveals in terms of perverse incentives that exist in the science system. While there remains debate4 as to the scale of the problem, greater care is needed in the terminology used.
4. Initial work on the issue of reproducibility in research highlighted the challenges that exist in the field of biomedical research5. Attention remains focused on the biomedical and social sciences. This is partly a function of the methodologies used within these disciplines. The extent to which people working within these research areas perceive a ‘reproducibility crisis’ as evident differs across these disciplines6.
Causes of irreproducible research
5. Understanding why a ‘reproducibility crisis’ exists requires an understanding of the methodologies by which the ‘crisis’ has been identified (i.e. what does a ‘crisis’ consist of in this context, how could this be identified, and does this condition exist), and separately why is a particular piece of research not reproducible. Causes of the latter include both honest error and deliberate falsification or exaggeration. It is important to note the distinction, even if practically the difference can be difficult to distinguish, and the outcome may be the same7.
6. Work by organisations including the Nuffield Council of Bioethics, the Royal Society, Wellcome and UKRI over the past eight years have highlighted pernicious aspects of the research culture, and the effects that this can have on the quality of research. High levels of competition within the system, while driving excellence in some instances, can also encourage behaviours such as ‘rushing to finish and publish research, employing less rigorous research methods and
1 https://royalsociety.org/topics-policy/projects/research-culture/
2 https://royalsociety.org/topics-policy/publications/2017/research-integrity-statement/
3 https://royalsociety.org/topics-policy/ethics- conduct/topic/#:~:text=Research%20funded%20by%20the%20Royal,for%20dealing%20with%20research%20misconduct. 4 https://www.pnas.org/content/115/11/2628
5 https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124
6 https://www.nature.com/articles/533452a
7 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5244822/
increased corner-cutting.’8 Participants in the Nuffield Council of Bioethics 2013 report on Scientific Culture, reported that ‘publishing in high impact factor journals is still thought to be the most important element in determining whether researchers gain funding’ and that ‘the pressure to publish in these journals is resulting in some types of important research findings not being published or recognised, for example, research with negative findings or research that replicated or refutes others’ work.’9 As negative findings and replication tend to be cited less often there is less incentive for academic journals to publish these types of articles and so a vicious cycle persists10. Participants in the 2018 Royal Society Research Culture conference reiterated these concerns11.
7. The use of machine learning12 approaches within research pose a particular challenge for reproducibility, one that may increase with the use of these methods. So-called ‘black box’ models are created directly from data by algorithms. Consequently, the people that design and use these models may not understand the process by which these models’ outputs are generated. Additionally, there can be issues in terms of generalisability across different datasets. Reproducibility requires that researchers can articulate the processes by which results have been generated. Explainable AI approaches seek to mitigate this, by helping ‘researchers to understand the insights that come from research data, by providing accessible interpretations of how AI systems conduct their analysis.’13
The role of the different actors in the research system
8. All actors in the research system have a responsibility to promote and support excellence. The revised Concordat to Support Research Integrity, published in 201914, illustrates the roles and responsibilities of individual researchers, employers of researchers and funders of research. Respectively, these include, ‘understanding the expected standards of rigour and integrity relevant to their research’, ‘maintaining a research environment that develops good research practice and embeds a culture of research integrity’, and ‘[taking] research integrity into account in the development of policies and processes.’15
9. In 2017, the Royal Society published its own ‘research integrity statement’. This sets outs expectations for ethics and integrity of the Society’s Fellowship and those in receipt of funding and is aimed at both individual researchers and institutions. The statement includes the expectation that, ‘researchers should conduct and communicate their research in a manner that best enables it to be reproduced.’16
10. Publishers, including the Royal Society, have a responsibility to support transparency and reproducibility in scientific research. In 2012, the Society published ‘Science as an open enterprise’17. This report defines open science as, ‘open data (available, intelligible, assessable and useable data) combined with open access to scientific publications and effective communication of their contents.’ In brief, the recommendations of this report were designed to, ‘improve the conduct of science, respond to changing public expectations and political culture and enable researchers to maximise the impact of their research. They are designed to ensure that reproducibility and self-correction are maintained in an era of massive data volumes.’18 Following the publication of this report, the Society’s journals introduced an open data policy that made the availability of data, code and research materials a condition of
8 https://www.nuffieldbioethics.org/assets/pdfs/The-culture-of-scientific-research-report.pdf
9 https://www.nuffieldbioethics.org/assets/pdfs/The-culture-of-scientific-research-report.pdf
10 https://royalsociety.org/news/2015/05/royal-society-open-science-to-tackle-publication-bias/
11 https://royalsociety.org/-/media/policy/projects/changing-expectations/changing-expectations-conference-report.pdf 12“Machine learning is a branch of AI that enables computer systems to perform specific tasks intelligently. Traditional approaches to programming rely on hardcoded rules, which set out how to solve a problem, step-by-step. In contrast, machine learning systems are set a task, and given a large amount of data to use as examples (and non-examples) of how this task can be achieved, or from which to detect patterns. The system then learns how best to achieve the desired output.” https://royalsociety.org/-/media/policy/projects/explainable-ai/AI-and-interpretability-policy-briefing.pdf
13 https://royalsociety.org/-/media/policy/projects/explainable-ai/AI-and-interpretability-policy-briefing.pdf
14 https://www.universitiesuk.ac.uk/sites/default/files/field/downloads/2021-08/Updated%20FINAL-the-concordat-to-support- research-integrity.pdf
15 https://www.universitiesuk.ac.uk/sites/default/files/field/downloads/2021-08/Updated%20FINAL-the-concordat-to-support- research-integrity.pdf
16 https://royalsociety.org/~/media/policy/Publications/2017/royal-society-research-integrity-statement-09-10-2017.pdf
17 https://royalsociety.org/~/media/royal_society_content/policy/projects/sape/2012-06-20-saoe.pdf
18 https://royalsociety.org/~/media/royal_society_content/policy/projects/sape/2012-06-20-saoe.pdf
publication.19 Other initiatives by the Royal Society journals to support openness and reproducibility include the introduction of open peer review, open identifiers for authors and the Accountable Replication Policy2021. The introduction of a reporting form published alongside journal articles by the Society’s journal Open Biology in 2018 is another example of practical action. The form requires the article author to disclose details including experimental design, source data, images and software used in their study22. The contents of the form are based on the NIH Principles and Guidelines for Reporting Preclinical Research23. The introduction of the form follows the launch of Registered Reports by Royal Society Open Science in 201524.
Policies and schemes that could have a positive impact on academia’s approach to reproducible research
11. Since the publication of Ioannidis’ 2005 paper, ‘Why Most Published Research Findings Are False,’25 several initiatives and innovations have been developed within the system to support research integrity and the reproducibility of research. Examples within the UK include the UK Reproducibility Network (UKRN)26 and ReproducibiliTea27. UKRN seeks to better understand the challenges that exist with regard reproducibility, and delivers training intended to disseminate best practice among researchers. Professor Marcus Munafo of Bristol University, who leads UKRN, was the lead author of the ‘manifesto for reproducible science’ published by Nature in 2017. Its recommendations included the promotion of transparency and open science28.
12. Work that looks beyond research metrics in the assessment of researcher performance, including the San Francisco Declaration on Research Assessment (DORA)29 and the Résumé for Researchers3031 can also play a positive role in reducing incentives to misrepresent research results.
13. As noted above, the Royal Society has promoted open science as a publisher and through its policy work. Action by funders, including UKRI’s Open Access Policy, help to support access and transparency and could have a positive impact on academia’s approach to reproducible research.
14. It is important to ensure that there is appropriate support and incentives for researchers to publish data and maintain datasets. Journal editors should normally ensure that data is being made available to other researchers in its original form, or via appropriate summary statistics where sensitive personal information is involved – including for published research conducted in the private sector32. Public research funders should better fund data collection and data sharing should be incentivised through appropriate recognition by the research community33. Research funders should in particular ensure that data handling, including the cost of preparing data and metadata, and associated costs, such as staff, is supported as a key part of research funding, and that researchers are actively encouraged across subject areas to apply for funds to cover this. Research funders should ensure that reviewers and panels assessing grants appreciate the value of such data management34.
15. In 2018, the Royal Society published the ‘Integrity in practice toolkit’ with the UK Research Integrity Office (UKRIO)35. UKRIO provides guidance and support to the public, researchers
19 https://royalsociety.org/journals/ethics-policies/data-sharing-mining/
20 https://royalsociety.org/blog/2018/10/reproducibility-meets-accountability/
21 https://royalsocietypublishing.org/rsos/replication-studies
22 https://royalsociety.org/blog/2018/07/transparent-reproducible-research/
23 https://www.nih.gov/research-training/rigor-reproducibility/principles-guidelines-reporting-preclinical-research
24 https://royalsociety.org/blog/2018/07/transparent-reproducible-research/
25 https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124
26 https://www.ukrn.org/
27 https://reproducibilitea.org/
28 https://www.nature.com/articles/s41562-016-0021
29 https://sfdora.org/
30 https://royalsociety.org/topics-policy/projects/research-culture/tools-for-support/resume-for-researchers/
31 https://www.ukri.org/apply-for-funding/how-were-improving-your-funding-experience/introducing-a-better-way-for-you-to- evidence-your-contributions/
32 https://royalsociety.org/-/media/policy/projects/dynamics-of-data-science/dynamics-of-data-science-skills-report.pdf
33 https://royalsociety.org/-/media/policy/Publications/2020/20-12-10-national-academies-national-data-strategy-response.pdf
34 https://royalsociety.org/-/media/policy/projects/machine-learning/publications/machine-learning-report.pdf
35 https://royalsociety.org/-/media/policy/projects/research-culture-images/integrity-in-practice-september-2018.pdf
and institutions on best practice in research integrity36. The aim of the toolkit is to inspire creative thinking in this area. It features a variety of new ideas from across the UK and the rest of the world, which require minimal to major resource support and can be led by individuals as well as institutions. The toolkit contains seven ways that could help individuals and institutions bring life to the codes, concordats and pledges to improve research integrity and culture that they have signed up to. The toolkit suggests additional mechanisms for establishing rigour and reproducibility including greater emphasis on reproducibility within faculty promotion processes37.
16. In addition to greater support for established methods to promote reproducibility as well as review of their efficacy, it is valuable to explore the role that new approaches and technological solutions could play. Specifically, some researchers have suggested that, while not relevant to all stages of the research process, blockchain technologies could make parts of the research cycle open to immutable provenance tracking and audit.38
17. Finally, The Royal Society will soon be publishing a report about how the online information environment affects discussions about scientific topics. It provides recommendations for improvements and explores subjects relevant to the Committee’s inquiry such as the manipulation of machine learning datasets, establishing provenance online, and researcher access to social media data. We are happy to provide a copy of the report to the Committee when it is published and meet to discuss its findings.
UK National Committee on Research Integrity
18. The UK National Committee on Research Integrity (CORI) can play a positive role in promoting reproducibility of research in the UK. The Royal Society supports the Concordat to Support Research Integrity39. As suggested earlier in this response, UK CORI’s role as a convenor and work to, ‘build and communicate the evidence base around UK research integrity’40 will be valuable. Specifically, there remains an issue around the transparency of reporting of incidences of misconduct. Investigations tend to be dealt with an institutional level. While it may be reported that an incident has occurred, the details of these investigations are not always shared with funders. Where this information is shared, there is scope for better sharing of information between funders. Given the current pressures on individuals and institutions, UK CORI will need to be mindful that it delivers its duties in a manner that relieves burden where possible.
(September 2021)
36 https://ukrio.org/
37 https://www.nature.com/articles/549133a
38 https://blogs.lse.ac.uk/impactofsocialsciences/2016/07/21/could-blockchain-provide-the-technical-fix-to-solve-sciences- reproducibility-crisis/
39 https://royalsociety.org/-/media/grants-schemes-awards/policy-and- positions/TheConcordatToSupportResearchIntegrity.pdf?la=en-GB&hash=46693DBE610B533841EB01CEF1154A2C 40 https://www.ukri.org/news/promoting-research-integrity-across-the-uk/