Written Evidence Submitted by the Centre for Education and Youth
(RRE0048)
The Centre for Education and Youth
The Centre for Education and Youth (CfEY) is a think and action tank, and our mission is to support all young people to make a fulfilling transition to adulthood. As a team of educational researchers, we closely monitor trends within the sector and reflect on how to the field can improve its research practices. The lead author on this submission, Baz Ramaiah, has written several pieces on the metascience of educational research.
The Reproducibility Crisis in Education Research (and why it matters)
Education has undergone an “evidence revolution” in the past decade. Teachers and school leaders are more interested than ever before in exploring educational research to find out ‘what works’. This interest has manifested as a rapid proliferation in blogs, journal clubs, conferences and courses in evidence-based teaching practice. All produced by teachers, for teachers.
However, the evidence base that teachers draw on remains compromised. Educational research still suffers from a lack of reproduction of findings. Section 1 of this submission summarises current research on this lack of reproduction, section 2 postulates causes and section 3 sets outs current and future solutions.
Reproducibility is one of the cornerstones of robust research. If a finding cannot be independently replicated, then it was likely a chance one-off yield, the product of methodological error or questionable research practices. Teachers who consume research need to be given confidence that a strategy or educational programme will work in their own setting, not just in the circumstances of the original study.
Reproduction of findings in educational research remains rare. A 2014 systematic review found that only 0.13% of articles published in the top 100 educational research journals are replications. To put that into context, the field of psychology was plunged into a “Replication Crisis” when its reproducibility index was found to be 1.1%.
The importance of replication studies is underscored by the findings they yield when they are conducted. The 2014 study discussed above found that of the few replication studies that were conducted within education, nearly half are unsuccessful. Similarly, recent research by the Education Endowment Foundation has failed to replicate high-profile findings that supported the effectiveness of particular educational programmes. For example, a 2021 replication of a 2016 study into the effectiveness of Philosophy for Children in improving pupils’ literacy and numeracy failed to produce the positive findings of the original study.
These null results remind us that, without routine replication, teachers are left with an unreliable evidence base to draw on when trying to improve their practice.
2. Causes of reproducibility in Education Research
There are several barriers to integrating replications studies into the education’s research culture. Challenges exist at three tiers:
Systemic challenges
Educational journals favour novel studies, rather than replications. Journals also continue to favour the publication of positive findings with large effect sizes, making it difficult to get negative yield replication studies published. As educational researchers are just as hostage to ‘publish or perish’ as any scientist, this disincentivises them from conducting replication studies.
Educational research journals have also still not adopted ‘Open Science’ practices that are becoming normalised in adjacent social sciences. Practices such as open publication of data from studies (for other researchers to quantitatively replicate by conducting another independent analysis) or providing detailed methodologies that support reproduction remains uncommon.
Some also argue that instruction for trainee researchers often fails to communicate the value of replication. As such, researchers enter the field having internalised the ethic of seeking novel results that they have observed in their seniors.
Methodological challenges
Most educational research takes place in schools. Schools are deeply complex places, with overlapping mosaics of individual, social, cultural and economic factors. Even within a school, these factors can vary heavily between classrooms and between days.
Because of this “variable-richness”, educational research findings are difficult to replicate. When the 2015 wave of replication studies in social and personality psychology began to yield null results, several researchers speculated that this was because psychology studies often have “hidden moderators”. These hidden moderators are the social, cultural and environmental context of a study. Because there are often such large volumes of these moderators, researchers cannot exhaustively detail them in their reporting of their findings. Even if they were able to, it is likely impossible to control for these as part of the study.
This means that research that takes place in schools may resist successful replication. As such, results in educational research may be ‘hyper-local’, only relevant to very specific circumstances.
Individual challenges
Even when replication studies are conducted and published, they suffer from dissemination bias. Such studies are typically less likely to be cited than the original study they were replicating, even in evidence summaries of relevant topic. Individual teachers demonstrate a strong attachment to their preferred approaches towards teaching. Even if they encounter a finding that invalidates the research finding their instructional approach is based on, confirmation bias means they are still likely to ignore it. Because of the autonomy teachers have within their own classroom it can be very difficult to shift teacher behaviour, even to align it with the best available research.
Some progress has already been made towards improving reproducibility in education. However, more work still needs to be done. We discuss pathways for the three classes of challenge described above.
Solutions to Systemic Challenges
Educational research journals should adopt open science practices that allow them to support future reproduction of published results. They should also amend their editorial and peer review practices to support the increased publication of replication studies. For example, making peer review results-blind and blind to whether a study is a replication may increase their publication rate in the field.
Researchers should also be made aware of open science practices such as pre-publication, which can be used as a means to draw attention to studies and findings (e.g. negative yield replications) without having to wait for changes to journal practices.
Education funders should support this by introducing constraints on funding that support future replication. The Education Endowment Foundation (EEF) in the UK have already made excellent efforts in this by requiring transparent, publicly available research protocols for trials. The EEF also do excellent work in funding replication studies and are committed to the publication of negative results.
Postgraduate training programmes for educational researchers should also include an emphasis on open science practices and replication as part of their instruction.
Solutions to Methodological Challenges
Researchers should be encouraged to practice ‘preproducibility’. This requires researchers to provide detailed and extensive accounts of the context they conduct research in and the decisions that they made as part of the research process. Funders and universities should incentivise this process by allowing researchers to price the auditing and write-up necessary for preproducibility into grant applications.
Solutions to Individual Challenges
Courses and content aimed at improving teacher’s engagement with and literacy of research should emphasise the importance of replication. This is already included in the Chartered College of Teaching’s course materials on interpreting evidence but needs to be adopted by other organisations.
Conclusion
Since the early days of the ‘replication crisis’, science has advanced rapidly in building the consensus and institutions required to support better research. At the same time, teachers have increased their engagement with educational research, drawing on it more and more as a guide for their practice. It is vital that educational research keep pace with science to ensure that teachers are in a position to give our young people the best education possible.
(September 2021)