Written Evidence Submitted by The National Physical Laboratory

(RRE0057)

Summary

-          Research funders and public funding bodies should require proposers to explain – in a separate, dedicated section in the proposal pro-forma – how they will ensure the quality and reproducibility of their research outputs using internationally recognised solutions.

-          Publishers should request that submissions to their journals prove how authors demonstrated the quality and reproducibility of their results and provide evidence of this.

 

 

Context to NPL’s response

  1. The National Physical Laboratory (NPL) is the UK’s National Metrology Institute (NMI), responsible for developing and maintaining the nation’s primary measurement standards on which all measurements rely. NPL is owned and funded (in part) by The Department for business, energy and industrial strategy (BEIS). NPL is a Public Sector Research Establishment (PSRE), which works in partnership with government, academia, applied research laboratories and industry to deliver the maximum societal and economic benefit for the UK and the world.
  2. NPL sits at the heart of the UK's National Measurement System (NMS) which maintains and develops the UK’s national measurement infrastructure and delivers the UK Measurement Strategy on behalf of BEIS. As the UK’s NMI we represent the UK within the international network of national metrology institutes globally that ensures the stability and comparability of measurement worldwide. 
  3. The national measurements standards NPL maintains sit at the top of the UK Quality Infrastructure (UKQI). UKQI is the system comprising the organizations (public and private) together with the policies, relevant legal and regulatory framework, and practices needed to support and enhance the quality, safety and environmental soundness of goods, services, and processes. It relies on metrology (the science of measurement), standardization, accreditation, conformity assessment and market surveillance to ensure reproducibility in measurement. NPL works at the interface of industry and academia, supporting researchers with reference materials and methods and leadership of inter-laboratory studies, which are a powerful tool for researchers to assess and improve their measurement performance. NPL’s post-graduate institute provides training on metrology to 198 students at 33 universities and helps equip them to be ‘truth-seekers’ in the research environment.
  4. Below we set out NPL’s responses to the topics that we consider most relevant to its area of expertise. 

 

The breadth of the reproducibility crisis and what research areas it is most prevalent in

  1. Most measurements made in the UK in support of regulatory compliance, fair trade and product conformity are required by law to be governed by the UK quality infrastructure to ensure consumer safety. This is the mechanism by which the UK ensures measurements in support of these critical requirements are reproduceable, have transparent processes, are of sufficient quality and are delivered with a fit for purpose level of confidence in the data produced.
  2. Most industrial research and development processes also employ similar systems to ensure the quality and reproducibility of their results, not because this is mandatory, but because it ensures the fit for purpose products and services are provided to customers effectively, efficiently, safely and at minimum cost. It also ensures that innovations can be brought to market as quickly as possible.
  3. Measurements underpinning research in academia do not have the same drivers as measurements in support of regulatory compliance or in product development. For instance, there is usually little requirement in grant proposals, and very rarely any local policy, requiring academic researchers to describe the systems they have used to assure the transparency, quality and reproducibility of their results. While in some disciplines there is a supportive research culture, for example in life-science much attention is paid to robust study design, sufficient statistical power and identification of confounding contributions, generally, there is no requirement for research publications to state the confidence with which conclusions are made.
  4. Very often lack of requirement to state confidence in conclusions is because the research output is itself the end point. Unless the work is subsequently taken forward to generate a product or service there is no incentive for the academic researchers to ensure alignment with the quality infrastructure. However, we know that research is taken forward, usually by people not involved in the initial work – this is how progress in science, and consequently society, is made – and who would benefit from an understanding of the quality of the outputs.

A greater attention to quality and reproducibility at the academic discovery science phase would:

  1. It is important to note that lack of reproducibility of research in itself may not be a problem. If all efforts to address data quality have been made and there is still lack of reproducibility observed, either within a research group or between different research groups, this is a beneficial outcome as it means a new influence parameter has been observed – an “unknown unknown” has been uncovered. This allows us to observe science through a stronger lens. Conversely, a lack of reproducibility without an assurance of measurement quality, blurs the lens through which we scrutinise science.
  2. The problem in some areas of scientific research, therefore, it is not the lack of reproducibility per se, it is the lack of reproducibility without assurance of measurement quality and transparency of process. Full transparency and the availability of open, FAIR data[1] in the publication of research work is a necessary but not sufficient requirement to address these issues. There also must be a strong focus on measurement quality and reproducibility through the entire process leading up to publication.

 

The issues in academia that have led to the reproducibility crisis

  1. The value chain in academia usually means that publication of results in peer-reviewed journals is an end in itself. Indeed, publication metrics are often the way success of an academic career is judged and so there is pressure to publish. This has led to some poor research practices and a preference for the publication of positive results. Whilst this culture is slowly changing – for instance via initiatives like DORA (sfdora.org), FAIR (go-fair.org), the UK Reproducibility Network (ukrn.org), and the acceptance of work for publication based on agreed methods rather than the results they produce –  there is still little incentive for academics to ensure the quality and reproducibility of the outputs of published work. Ultimately it is this that must change if the reproducibility crisis is to be properly addressed – it is the root cause of most reproducibility problems and the issue to which the least attention has been paid currently. It will not be resolved completely by any of the initiatives currently proposed or in place.
  2. A necessary, but not sufficient, requirement for reproducibility in science is ensuring all data is openly available. The FAIR data principles require researchers to provide relevant background information (metadata) to give their data context. This information generally covers how the data were collected (standard followed, device used, environmental conditions, etc.), when they were collected, and by whom. This contextual information ensures that anyone wishing to reproduce the experiment has access to the values of the various factors that may affect the experimental outcome, which is one required aspect of improving reproducibility.
  3. There are, however, several limitations to the FAIR principles. The first is that they explicitly do not address data quality: published data can satisfy the FAIR principles and still be of poor quality and (which is worse) not be identified as such. The second limitation is that the background information supplied with the data is generally not subject to standardisation and is therefore a subjective reflection of what any given researcher believes to be important. Some areas of research do have metadata standards, either official or de facto, but many areas do not, and so published metadata may fall short of what is required to ensure reproducibility. A third potential problem is metadata quality. Metadata capture is generally a secondary consideration to researchers, and in many cases the information is likely to be captured by hand (whether on paper or electronically) and hence potentially prone to errors and omissions. Wider usage of approaches to automated metadata capture would address this problem.
  4. Various characteristics are used to describe data quality, depending on the intended use of the data. Example characteristics include accuracy, completeness, consistency, reliability, and relevance. For scientific research, accuracy and reliability are the most difficult for an external reviewer to assess and are strongly dependent on information supplied by the researcher. A measurement quality process provides a framework to quantify and report accuracy and to demonstrate reliability through traceability back to national standards, and hence to improve reproducibility.

 

The role of the following in addressing the reproducibility crisis

  1. Pressures in all research environments (academia, industry, national laboratories) can unintentionally encourage ‘progression-seeking’ behaviour over ‘truth-seeking behaviour’ [1]. These terms are used in the pharmaceutical industry, who were amongst the first to highlight the issues of a lack of reproducibility [2-4]. A truth-seeking culture is essential to improve reproducibility. Progression-seeking is where the desire to meet an objective, whether a high-profile paper or an innovative product, overrides the maybe weak warning signals which should trigger caution and the need for further investigation – truth seeking.

 

Research funders, including public funding bodies

  1. Research funders and public funding bodies should require proposers to explain – in a separate, dedicated section in the proposal pro-forma – how they will ensure the quality and reproducibility of their research outputs using internationally recognised solutions [5]. For example: using agreed standard methods, validation of new methods, ensuring traceability to internationally agreed references, gaining accreditation, and ‘pathways to reproducibility’ [5]. This should be given a high weighting on the assessment of proposals and properly monitored during the lifetime of successful proposals. It would be sensible to engage measurement quality specialists, such as those at NPL, in the assessment and marking of these sections.

 

Research institutions and groups

  1. Research institutions and groups must respond to changes required in demonstrating the quality of outputs by challenging internal cultures and seeking training in routes to measurement quality and reproducibility. A truth-seeking culture needs to be embedded. More emphasis needs to be given to include step-by-step protocols for repeating experiments to go alongside publications.

 

Individual researchers

  1. Individual researchers must take responsibility for the quality and reproducibility of their own research outputs and encourage colleagues to do the same. Researchers should be aware of what progression-seeking pressures they are under and manage them appropriately. They should watch out for outliers that contradict expected results and ensure that everyone in the research team understands the importance of reproducibility of research. High-quality metadata should be recorded to enable cross-checking of instruments (calibration status, recent modifications etc) as well as sample details and history.

 

Publishers

  1. Publishers must recognise that open data and transparent publishing is not the complete solution. It is a necessary step in addressing reproducibility in science but alone is not sufficient. Publishers should request that submissions to their journals include how authors demonstrated the quality and reproducibility of their results and provide evidence of this.
  2. Authors should also be asked to provide a level of confidence with which they draw their conclusions. This is commonplace in traditional measurement science, but seldom seen in other research communities. One way of approaching what is a novel concept for many might be to provide an agreed scale (as is sometime the case for risk description in medical studies), as an example: very high confidence (99 %: very likely to be true), high confidence (95 %: likely to be true), etc, as outlined in annex G of the guide to the expression of uncertainty in measurement [6]. 

 

What policies or schemes could have a positive impact on academia’s approach to reproducible research?

  1. The work of independent organisations such as the UK Research Integrity Office and the UK Reproducibility Network have been very useful in raising the profile of these issues, especially reproducibility in science. However, to date these have not necessarily been targeted at improving the quality of published research data. This needs policy action, in particular a requirement for publicly funded research not only to make all data and methods available but also to provide demonstration of the quality of the data produced and evidence of the confidence in the conclusions drawn.
  2. The concordats to support research integrity and open research data are raising awareness, encouraging best practice and are generally having a positive impact. An increased awareness of Research Culture [7], exposure of bad practice and recommendations for good practice are also making important improvements. NPL uses its expertise to provide validation and ensure traceability and quality to methods and research outputs.

 

How establishing a national committee on research integrity under UKRI could impact the reproducibility crisis

  1. A national committee on research integrity established under UKRI could have a beneficial effect on UK reproducibility in science if a major part of the committee’s remit was to address the quality and reproducibility of data, as highlighted in this response.

 

References

[1] Ringel, M., Tollman, P., Hersch, G. et al. Does size matter in R&D productivity? If not, what does?. Nat Rev Drug Discov 12, 901–902 (2013). https://doi.org/10.1038/nrd4164

[2] Prinz, F., Schlange, T. & Asadullah, K. Believe it or not: how much can we rely on published data on potential drug targets?. Nat Rev Drug Discov 10, 712 (2011). https://doi.org/10.1038/nrd3439-c1

[3] Begley, C., Ellis, L. Raise standards for preclinical cancer research. Nature 483, 531–533 (2012). https://doi.org/10.1038/483531a

[4] Reproducibility and reliability of biomedical research: improving research practice. Symposium report, October 2015 https://acmedsci.ac.uk/file-download/38189-56531416e2949.pdf

[5] Sené, M., Gilmore, I. & Janssen, JT. Metrology is key to reproducing results. Nature 547, 397–399 (2017). https://doi.org/10.1038/547397a

[6] Guide to the expression of uncertainty in measurement, JCGM 100:2008, BIPM, https://www.bipm.org/documents/20126/2071204/JCGM_100_2008_E.pdf/cb0ef43f-baa5-11cf-3f85-4dcd86f77bd6

[7] What Researchers Think About the Culture They Work In Wellcome Trust, https://wellcome.org/sites/default/files/what-researchers-think-about-the-culture-they-work-in.pdf

 

 

(September 2021)

 

[1] https://www.go-fair.org/fair-principles/