Written Evidence Submitted by the Department of Psychology, University of York

(RRE0010)

We are academics from the Department of Psychology, University of York. Dr Hobson is a Lecturer with research interests in language and communication difficulties. Dr de Bruin is a Lecturer with research interests in bilingualism. Dr Horner is a Senior Lecturer with research interests in memory. We have been involved in promoting replication and reproducibility at the departmental and university level, have engaged with wider community-led initiatives to improve reproducibility in psychology, and have experience in curating and sharing data and publishing registered reports.

To ensure clarity in our answers, we provide definitions for reproducibility and replication. Whereas reproducibility relates to whether the results of an experiment can be reproduced based on the same data (i.e., if you reran the same analysis script on the same data set, would you obtain the same result?), replication refers to finding a similar result in a separate data set. Ensuring research is reproducible requires data sets and analysis scripts to be open and accessible to enable other researchers to reproduce the original results. Replication requires sufficient methodological detail to enable other researchers to carry out a replication under the same conditions as the original experiment. Both reproducibility and replication are made easier in an environment where open research, under the FAIR principles (https://www.nature.com/articles/sdata201618), is encouraged.

Before we discuss our views regarding areas of improvement, we would like to emphasise that the past years have seen many new initiatives related to open research and reproducibility. At The University of York, we have an Open Research Strategy Group (https://www.york.ac.uk/staff/research/open-research/open-research-statement/) as well as a team of “open research advocates” (https://www.york.ac.uk/staff/research/open-research/). In these groups, researchers across disciplines and library staff meet regularly to discuss open research practices. This is in addition to an “open research team” who support researchers and provide training in open research practices, including reproducibility. Within the Department of Psychology, we have an active “Open Research Interest Group” (https://www.york.ac.uk/psychology/research/groups/open-science/) that provides further training and discussion. As part of this group, we ask PhD students, postdoctoral researchers, and staff to complete an annual survey regarding their open research practices. This shows a clear increase in the number of people adopting open research practices (all of which are important for reproducibility), including publication of pre-prints (55% of the respondents in 2020 had experience doing this versus 26% in 2017), data sharing (62% in 2020 versus 58% in 2017), and pre-registering the study plans and designs before data collection (52% in 2020 versus 6% in 2017). Open research practices and awareness of the need for reproducible research are thus becoming more common. However, the change in research culture that is required for these practices to become the norm takes time. Below we outline some ways in which these changes in culture can be supported, including examples of initiatives that have been set up in the UK and internationally. Our key point is that bottom-up change is occurring and needs support to ensure long-lasting change.

1)               The breadth of the reproducibility crisis and what research areas it is most prevalent in

The reproducibility crisis is an issue that touches on all biomedical and social sciences, although the documentation of this is not equal across the fields. Psychology is a field in which the issues have been widely discussed (see the report by the Open Science Collaboration "Estimating the reproducibility of psychological science", 2015), which may create the sense that it is an area of specific concern for this field. However, the same features and pressures operate across disciplines and subdisciplines, meaning the academic ecosystem that leads to reproducibility problems is similar across social and biomedical sciences in the UK.

2)               The issues in academia that have led to the reproducibility crisis;

As has been outlined by academics such as Prof Marcus Munafo, and the Academy of Medical Sciences’ report on Reproducibility and Reliability of Biomedical Sciences (available here: https://acmedsci.ac.uk/policy/policy-projects/reproducibility-and-reliability-of-biomedical-research), the incentive structures in academia promote fast, high impact, and novel results, but disincentivise slower, more careful research. Replications – a key process by which research findings are tested and checked – are commonly considered “boring”, are harder to publish, are less likely to be accepted by high impact journals, and are overall not rewarded in the same manner as being the first to publish an effect. This effect carries through not only when academics attempt to publish their work, but also when seeking to gain funding for their research: research proposals are expected to be ground-breaking and novel. They are not expected to consolidate, replicate and ensure the reproducibility of an effect. This has also been reflected in the REF (Research Excellence Framework) exercise, which, while it mentions “rigour”, also calls for “originality” and “significance” . As noted in the sections below, this research culture may retain individuals who succeed in such a system, but those who are frustrated with these issues may leave research, perpetuating a culture of these practices.

3) The role of the following in addressing the reproducibility crisis:

a) Research funders, including public funding bodies

As outlined above, research funders largely call for highly novel approaches. A research proposal that is a series of well-planned replications is unlikely to be funded. Similarly, a research proposal that proposes novel experiments followed by replication experiments would likely score lower than one proposing a greater number of novel experiments without efforts to replicate. However, it is worth noting that the reviewers of such bids are themselves individual researchers (typically), and thus this issue also pertains to the attitudes and behaviours of individual researchers working on behalf of a funding body. Funding bodies therefore need to take reproducibility and replication seriously, and should have clear guidance for grant reviewers and panel members to encourage change.

In addition to guidance, separate grants could be set up specifically for studies that aim to reproduce and/or replicate findings. The Dutch equivalent of UKRI (NWO) launched a “pilot programme Replication studies” in 2016 (https://www.nwo.nl/en/researchprogrammes/replication-studies). This programme funded studies that aimed to reproduce existing data sets, replicate with new data following the protocol of the original study, or replicate with a different protocol but the same research question as the original study. The aim was to make replication research more common without creating competition with existing grant schemes. The scheme was set up in response to a report “Improving reproducibility in empirical sciences”, written by the KNAW (Royal Netherlands Academy of Arts and Sciences; their full report  can be found here: https://www.knaw.nl/en/news/publications/replication-studies?set_language=en)

Ensuring research is reproducible can be time consuming. For example, a key aspect of increasing reproducibility is making sure your analyses and code are correct, properly archived, and accessible. Asking colleagues to give feedback on your analysis and code is an important step in this process but takes time and expertise. To facilitate these practices and increase reproducibility, researchers could include staff in grant applications who help with tasks such as programming experiments in ways that allow other researchers to use the same materials and create reproducible analyses and scripts. This practice could be explicitly encouraged by research funders.

b) Research institutions and groups

A key factor in the promotion of university staff is their ability to publish papers and secure grant income. If methods that align with non-reproducible/replicable research (e.g. small sample sizes, over-analysing data) are the methods most likely to achieve fast, high impact publication and secure grant income, then individual researchers will be inclined to adopt them. The recruitment and promotion process therefore encourages practices that may not align with practices that promote reproducible and replicable research. Institutions have the potential to contribute to this issue by changing recruitment and promotions criteria.

Research institutions and groups can help their staff by encouraging open discussions about reproducibility and replication studies. Setting up peer-support groups (such as the ReproducibiliTea network, https://reproducibilitea.org/) helps researchers to share best practices and to discuss issues that might arise when trying to improve reproducibility. Within our department, this has created an informal and accessible peer-support system. Ideally, however, such a support system would be funded and/or guided by support staff who are hired specifically to help with open research matters. Without this support, there is a risk that researchers (often early-career researchers) are overburdened with requests for help, while often not receiving the credit (e.g., co-authorship) that is needed for them to progress in their career.

c) Individual researchers

Funding bodies, Universities, and publishers are largely staffed by individual researchers, or individuals who have a research background. Individual researchers, if motivated to do so, therefore have enormous power to shift attitudes and processes: individual researchers could for example choose to review less favourably grant submissions that do not propose reproducible approaches/rate more highly those that include steps such a replication, suitable power analyses etc.

It is worth noting that the power individual researchers have is dependent on factors including career stage. Generally, the reproducibility movement is considered to have had good buy-in from Early Career Researchers. However, ECRs are often on time-limited contracts, lack their own source of funding, and need early publications/grants in order to secure a permanent position in academia and research. Therefore, even if motivated to change culture and practice, they may have limited means to do so. ECRs who want to engage in reproducible practices but are frustrated by the lack of opportunity to do so may be at risk of leaving research: therefore, we should be mindful of “survivor bias” in those who have remained in research careers.

There have been senior academics who have been very prominent in their advocacy for open research practices (e.g. from our own field: Prof Dorothy Bishop, Prof Chris Chambers, Prof Marcus Munafo). However, some established academics might be less likely to change practices, if they are not encouraged to do so, or do not receive support and resources (e.g. time for training) to adopt new practices. Given the “publish or perish” culture that does not reward slow and careful science, they may furthermore see it in their students’ best interests to secure publications, even if the methods lack reproducibility. Permanent changes in research culture requires consideration of not just ECRs (at whom many initiatives have already been targeted), but also senior academics.

d) Publishers

Publishers have begun to adopt some approaches that could help improve reproducibility. For example, some now offer researchers the ability to publish via registered reports. Under this approach, research articles are reviewed prior to data collection and thus cannot be reviewed on the basis of the findings, only the methodological quality of the study. However, such a publication route is by no means universal and some fields are lagging behind in their adoption. For example, Dr Hobson’s research includes research on the topic of autism: recently her group sought to register a study on the topic of autism and none of the major autism journals offered registered reports. A small group petitioned the journals to offer this publication route, and one journal is now in the process of offering this.

Another example of the changing landscape of publication is the concept of “data reports” at the Journal of Cognition (https://www.journalofcognition.org/about/submissions/#Article%20types). This offers researchers the ability to publish data sets that they believe to be of value to the community that might typically be difficult to publish if they were not associated with a novel finding. Dr Horner has recently used this format to publish a developmental data set that, although it did not provide a clear theoretical contribution to the literature, had value to the research community (https://www.journalofcognition.org/articles/10.5334/joc.149/).

It is important to note that journal formats such as registered reports and data reports should not be viewed as “better” or replacements for more typical formats. However, they offer much needed diversity in the publication landscape, allowing researchers to publish, gain credit for, and disseminate different aspects of their research to the wider research community. We strongly support efforts to increase this diversity, as opposed to prioritising one format at the expense of another.

e) Governments and the need for a unilateral response / action.

We believe that the government’s action should be carefully considered. Punitive action or poorly thought through messaging may undermine public trust in scientists and researchers. Public discussion around the issues of reproducibility will require clear and sensitive messaging . We cannot lose sight of the incredible advances made in the sciences. Improvements in reproducibility should therefore be seen as a means of improving on an already highly efficient, innovative scientific landscape in the UK. It is also important to note that a large proportion of research will always be non-replicable. Science is by its nature a journey into the unknown. We need to allow for an environment where mistakes can be made and that they not be punitive to those who made the mistakes.

One notable arm by which the government could already act is careful review of the REF (Research Excellence Framework), and what the REF is incentivising in UK Universities. Alignment of REF ratings with reproducible practices could help, although we would caution adding increased burden to academic/researchers via the REF (this could actually take away from time spent doing the research), and be mindful of what this re-alignment would mean for research that does not fit a traditional reproducibility framework (e.g. qualitative work).

4)     What policies or schemes could have a positive impact on academia’s approach to reproducible research;

Many ideas are already laid out in the Academy of Medical Sciences report (available here: https://acmedsci.ac.uk/policy/policy-projects/reproducibility-and-reliability-of-biomedical-research). Our view is that we will need a range of policies or schemes, and that any particular policy is unlikely to be the “silver bullet”, especially as different fields and subfields will have different needs and requirements. Funding that is specifically for replication would help (see e.g., the Dutch pilot scheme described in section 3a).

Encouragement for researchers to appreciate the difference between hypothesis generating and hypothesis testing research (i.e. discovery science versus consolidation), and to pre-register studies in which they are testing a clear hypothesis, would help. Embedding knowledge about the issues of reproducibility (and potential solutions, such as how to run suitable power analyses) in UK biomedical and social science degrees would also help. The British Psychological Society (BPS) have been discussing these ideas, as to receive accreditation by the BPS, Psychology degrees must include suitable statistical and research methods training. Making it a requirement that issues around reproducibility are covered in degree courses would ensure an awareness of the issues, and solutions, in the next generation of researchers. However, this must be coupled with training and incentives for senior researchers.

Ultimately, robust, replicable, and reproducible science is only possible in a well-funded, supportive, collaborative environment where scientists are rewarded for publishing fewer, better, papers. Incentivising reproducible and replicable science is not incompatible with ensuring the scientific community are pursuing novel, impactful, research topics. It is simply ensuring the evidence base in these topics is as strong as it could be.

5)               How establishing a national committee on research integrity under UKRI could impact the reproducibility crisis.

We think oversight of these issues, and how different fields are responding to them, would be excellent, but as noted above we believe it would be wise to be careful that establishing such a group would not lead to public distrust of researchers. There is already the UK Reproducibility Network (https://www.ukrn.org/), who would be well placed to support cultural change given the wide network of researchers that they have established.

Critically, any top-down effort to encourage more robust, reproducible, and replicable research needs to ensure that scientists are the driving force of such change, and that the focus is on supporting bottom-up change rather than imposing top-down structural changes that apply more pressure in an already highly-pressured profession. As documented above, there are already a wide range of bottom-up initiatives that are well embedded at the departmental, institutional, and interinstitutional level across the UK HE sector that have had a demonstrable impact on increasing replicability and reproducibility. The best way to ensure UK science prioritises reproducibility is by supporting such initiatives. Furthermore, it is crucial to keep in mind that changing a (research) culture takes time and that any chances are likely to only become visible in the long run. Any initiatives focused on short-term changes and fast progress are therefore unlikely to succeed in creating any real, long lasting changes.

A committee would need to consider:

●       The range of different fields and subfields, and that one approach to support reproducibility in one field may not be applicable in another;

●       The voice of researchers, so that the committee’s considerations are driven by and directly address the researchers’ needs.

●       How to embed an awareness and appreciation of reproducibility and reproducible methods in UK undergraduate and postgraduate degrees;

●       How work on the issue of reproducibility might be seen to devalue certain forms of research: for example, qualitative research does not fit into the traditional approach to reproducibility (although credibility and transparency are very important for qualitative approaches, and could be considered under a wider “research integrity” committee).

●       The alignment of research ideals (reproducible, careful, ability to be applied to real world practice) with research incentives (publications, grants, promotion tracks, REF funding), with consideration of how processes that seek to bring this realignment about could adversely impact research activities (e.g. increasing time on REF activities will detract from research time).

●       The wide range of existing initiatives to increase reproducibility across the UK HE sector, and how they can be best supported to encourage long-term cultural change.

(September 2021)