International Development Committee Inquiry: Leaving no one behind

Submitted by Birte Snilstveit (bsnilstveit@3ieimpact.org), Senior Evaluation Specialist, International Initiative for Impact Evaluation (3ie), on behalf of 3ie.

Summary

The evidence presented in this submission is based on the most comprehensive systematic review of evidence on education effectiveness in low and middle income countries conducted to date (Snilstveit et al., 2015; Snilstveit et al., 2016). We assessed the evidence on 21 different types of primary and secondary education programmes on children’s school enrolment, attendance, drop-out, completion and learning outcomes.

The evidence base assembled in this review include 238 impact evaluations, covering over 16 million children participating in 216 education programmes in 52 low-and middle-income countries (L&MICs). The findings from this study can help inform decisions about effective strategies for achieving the education targets:

Most education programmes typically improve either participation (school enrolment, attendance, completion and drop-out) or learning outcomes. They rarely improve both. This suggests that tackling the learning crisis requires concurrently addressing multiple barriers to quality education.

What works in most contexts: Cash transfers had the largest and most consistent positive effects on increasing school enrolment, reducing dropouts and improving completion. However, on average, they have not improved learning outcomes.

Structured pedagogy programmes had the largest and most consistent positive effects on improving learning outcomes. These are programmes that typically provide customised curricula, new instructional approaches and teachers’ training, and educational materials for students.

What is promising: School-feeding and community-based monitoring interventions are among the few programmes that are promising for improving school enrolment as well as learning outcomes.

Public-private partnerships, new schools and toilets are promising for improving participation outcomes. Extending the school day, remedial education programmes and merit-based scholarships are promising for improving learning outcomes.

What doesn’t always work: School-based management programmes, computer-assisted learning, programmes providing education materials have not improved learning outcomes in most contexts. In the case of computer-assisted learning, the effects on learning outcomes have even been negative in some contexts.

What is unknown: The overall effects of implementing school-based health programmes, providing information to children or parents and reducing user fees are not clear because there have been few high-quality studies on these interventions.

Conclusion: To address the global learning crisis requires a major course correction. Part of the solution will be directing more resources to the education sector. But more funding is not sufficient for addressing the learning crisis - resources need to be directed to programmes that work. This means that education decision makers need to have access to, and incentives to use, high quality evidence.

Introduction

We welcome the International Development Committee’s inquiry into DFID's work on education. The evidence presented in this submission responds in particular to the following two questions:

 

 

The evidence presented in this submission is based on the most comprehensive systematic review of evidence on education effectiveness in low and middle income countries conducted to date (Snilstveit et al., 2015; Snilstveit et al., 2016).

The review synthesised evidence on the effects of 21 different types of primary and secondary education interventions on children’s school enrolment, attendance, drop-out, completion and learning, specifically maths, language and composite test scores. We did not cover evidence on early childhood development.

The study was conducted using standards for high quality systematic reviews as per the Campbell and Cochrane Collaborations’ handbooks (Higgins & Green 2011; Campbell Collaboration 2015). Box 1 explains the categories used for interpreting the results of those analyses in terms of effectiveness.

In this note we summarise the main findings of our study. Further details are available in our systematic review summary report and technical report. We are happy to answer any further questions the Committee may have.

 

 

The evidence base

We identified and reviewed in detail evidence from 238 impact evaluations and 121 process evaluations/ qualitative studies. The studies cover over 16 million children participating in 216 education programmes in 52 low and middle income countries.

Figure 1 shows the breakdown of studies by intervention type. It highlights that the distribution of impact evaluations of education interventions is uneven across different areas. Interventions such as cash transfers, structured pedagogy and computer-assisted learning programmes have been studied quite extensively.

For other programmes, such as school-based health, information to children, teacher interventions, remedial education and school day extension, the evidence is more limited. This is significant because development assistance has been directed to many of these understudied areas.

Another important evidence gap is the lack of studies focusing on the most vulnerable and marginalised children, or providing analyses of effects by different sub-groups of children. This limits our ability to identify programmes that are more effective in targeting the hardest to reach children.

 

Figure 1: Distribution of studies is uneven across interventions

 

 

 

 

 

Main findings

A range of programmes can and have improved school participation and learning outcomes for primary and secondary school-aged children in L&MICs. There are no ‘magic bullets’ to ensure high-quality education for all, but there are lessons to be learned for improving future education programmes. Figure 2 provides a visual overview of the findings of our review.

Figure 2: Visual overview of results across the systematic review by intervention area

Our work highlights the following key issues in relation to the focus of this inquiry:

Programmes typically improve either school participation or learning outcomes, but not both: Looking across more than 20 different types of interventions we find that, with a few exceptions, programmes typically improve either school participation or learning outcomes, but not both. The exceptions are community-based monitoring, school-feeding and multi-component interventions. The evidence suggests these interventions have improved both school participation and learning outcomes in some contexts, although more evidence is needed to confirm this finding.

Cash transfer programmes and structured pedagogy interventions consistently improve access and learning respectively: Of all the interventions included in the systematic review, there is fairly strong and consistent evidence that cash transfer programmes result in the largest and most consistent improvements in school participation. Such programmes have increased school enrolment, reduced drop out and improved completion rates across a range of contexts in Latin America and the Caribbean, Sub-Saharan Africa, South and East Asia and Pacific, North Africa and Europe and Central Asia. On average, cash transfer programmes do not improve numeracy and literacy outcomes however, although some individual programmes show positive results.

Of the interventions included in the review, structured pedagogy programmes have the largest and most consistent positive effects on learning outcomes. Effects on participation outcomes for structured pedagogy programmes are not clear because few studies measure such outcomes. Most of these programmes took place in Sub-Saharan-Africa, but there were also several in South Asia, East Asia and the Pacific and Latin America. These programmes typically address several constraints to learning directly, such as lack of appropriate materials, curriculum and poorly trained teachers. In doing so they provide a carefully designed combination of new, appropriate content and materials for students, and materials and training for teachers in delivering the new curriculum.

A number of programmes are promising, but we need more research to confirm effectiveness: A range of other interventions have also improved outcomes substantially in some contexts and are considered promising (see figure 2). More research is needed to assess whether these improvements can be replicated elsewhere.

Community-based monitoring interventions improve school enrolment in some contexts, but effects on other participation outcomes (such as student attendance, completion and drop out) are less clear. The results also suggest community-based monitoring interventions can improve student test scores, but effects vary between contexts – even within the same country.

School-feeding programmes are one of the programmes included in this review that are promising for improving both school participation and learning outcomes, particularly in contexts with high rates of food insecurity and low rates of school participation. School-feeding programmes have limited effect on these outcomes in areas without malnutrition and where school participation rates are already high.

Public–private partnerships have improved school participation in some contexts. In particular, we find that public–private partnerships involving per student subsidies to low-cost private schools have had substantial effects on school enrolment rates in low-income settings, as replicated across three programmes in Haiti, Pakistan and Uganda.

Construction of new schools and programmes to construct latrines in schools also appear promising for improving school participation, with greater benefits for girls.

Merit-based scholarships (where students, rather than parents, are offered financial incentives to improve attendance and effort), remedial education, additional instructional time and construction of new schools are promising interventions for improving learning outcomes.

Some programmes do not work in all contexts: For some commonly implemented programmes, the results are less encouraging.

Programmes providing new learning materials had little or no effect on learning outcomes, when measured in terms of maths, language arts and composite test scores. Qualitative studies suggest several programmes suffered from implementation challenges and that the materials provided to students were not always appropriately customised. This kind of educational ‘hardware’ is a necessary, but not sufficient input for improving learning outcomes, especially if poorly designed or implemented.

Computer-assisted learning programmes have mixed effects on learning outcomes. These programmes can improve learning, especially in maths. However, results vary greatly by location, and the overall effects on language and composite test scores are close to zero. These programmes have also had harmful effects on children’s learning in contexts where they have been poorly designed and/or implemented.

Teacher incentive programmes appear to have small overall effects on teacher performance and student outcomes, based on the few available studies. The average effects range from small negative effects on some measures of teacher performance (use of materials) to no effects on a range of student participation outcomes (student attendance, drop out, completion) and learning outcomes (language arts, composite test scores). Maths test scores may be an exception, with modest overall improvements. Despite the popularity of such programmes, the evidence suggests that teachers do not seem to improve their attendance or alter their instruction techniques in response to incentives, with the exception of an increased focus on preparing for tests.

School-based management interventions do not appear to improve school participation, with little or no improvement, on average, in enrolment, completion and drop-out rates. This finding is consistent across programmes and outcomes, although few studies are available to assess the results for enrolment. Similarly, the overall effect of school-based management interventions on learning outcomes is close to zero. But there is a large amount of variability in effects across contexts, with larger improvements in learning outcomes in several programmes in the Philippines.

Tackling the learning crisis requires concurrently addressing multiple barriers to quality education: A cross-cutting finding of this study is that successful programmes address constraints at multiple levels. Children’s education outcomes are influenced by a range of factors, such as their health and nutrition, the socio-economic situation of their household, the materials and human resources available in the classroom and the management of these resources. It is thus not surprising that outcomes of any one intervention may be affected by the extent to which other major constraints remain unaddressed.

This is illustrated by programmes that seek to reduce schooling costs, such as cash transfer programmes. Cash transfer programmes were the most effective intervention in boosting school attendance and other participation outcomes. However, they had little effect on learning outcomes, as measured by maths, language and composite test scores. This could be due to the low quality of schools that those children attend. Unless cash transfer programmes are also accompanied by additional resources for schools and teachers, a sharp increase in the number of children enrolled or attending school may reduce the quality of education provided by schools.

Similarly, programmes that attempt to improve the governance and quality of education systems address barriers such as information asymmetries and lack of accountability between service users and providers. They do not address other constraints, such as a lack of trained teachers or insufficient school materials. Thus, in resource-poor contexts, school-based management and community-based monitoring programmes were not always sufficient to improve education outcomes.

Tailoring programmes to suit baseline constraints and capacities can improve the chance for success. Many of the successful programmes were those that were tailored to the human and social capital of the location in which they were delivered. School-feeding programmes, for example, had the largest effect in areas with high levels of food insecurity, malnutrition and low school attendance. The effects were much smaller in better-off areas where enrolment was already high and malnutrition was less common.

Social capital may be an important factor for interventions intended to improve school governance. School-based management was most successful in settings with high levels of social capital and a tradition of local participation. In the Philippines, where school-based management had consistently positive effects, the qualitative evidence suggests parents and communities were willing and able to make basic decisions about schooling when given the opportunity to do so. In contrast, results in most other contexts were disappointing. Evidence from Niger and the Gambia suggest that low social and human capital may limit the success of school-based management programmes.

 

Interventions with relatively short causal chains that directly target barriers to participation and learning may be more effective. Programmes that directly address constraints to improve an outcome appear more likely to succeed. Such programmes have fewer assumptions about how people and systems will respond, and the risk of implementation failure is reduced.

For example, if the main constraint to attendance is family resources, it is not surprising that cash transfers consistently improve attendance. The causal chain between cash transfers and their impact on learning is much longer: transfers would need to improve household income to the extent that the learning environment for the child is improved, and the child would then need to study harder and perform better in the classroom.

Similarly, structured pedagogy programmes provide a range of different components to address barriers to learning at the school and teacher level, such as lack of appropriate materials and qualified teachers. The causal chain between the programme and its impact on learning is relatively short.

Programmes involving teachers need to consider the time and motivation required for them to participate. Teachers are the most important actors of the education system. If teachers do not participate in programmes in which they have a key role, the programme cannot be successful. Thus, programmes need to consider their time and motivation. For example, evidence from some computer-assisted learning programmes suggests programme designs need to consider teachers’ workloads, and their attitudes and motivation for making radical changes in the way they teach. Evidence from interventions that are directly targeted at teachers also show that programmes need to carefully consider the incentives that affect teacher behaviour.

Currently, substantial resources are directed to programmes where effects remain unknown: There are important gaps in the current evidence base, which limits our ability to implement evidence informed programmes. While the volume of evidence identified in this exercise may seem large, when we consider the broad scope it covered in terms of types of programmes and number of studies, there are important evidence gaps. For example, the effects of school based health programmes, different models of teacher training and teacher hiring, user fee reduction and grouping students by ability are unknown. There is also very limited evidence on how to best target the most vulnerable children and reduce inequities in outcomes. Finally, for some regions, such as North and West Africa, and large, populous countries like Bangladesh, Indonesia and Nigeria the existing evidence is very limited.

Conclusions

The results from our systematic review provide several important findings to inform future DFID programmes. Firstly, the results demonstrate that programmes can improve school participation and learning outcomes in L&MICs. Secondly, looking across more than 20 different types of intervention we find that, with a few exceptions, programmes typically improve either school participation or learning outcomes, but not both.

Thirdly, some programmes are more effective than others. There is fairly strong and consistent evidence that cash transfer programmes have relatively large positive effects on school participation outcomes, while structured pedagogy programmes have the largest and most consistent positive effects on learning outcomes. A range of other interventions have also improved outcomes substantially in some contexts and are considered promising. Finally, to improve the chance of success it is important to tailor programmes to suit baseline constraints and local capacities.

To address the global learning crisis requires a major course correction. Part of the solution will be directing more resources to the education sector. But more funding is not sufficient for addressing the learning crisis. Resources need to be directed to programmes that work. This means that education decision makers, both in funding agencies and low and middle income countries, need to have access to high quality evidence and the capacity to use that evidence to inform decisions about policy and programming in their particular context.

But there are important gaps in the current evidence base and substantial resources are spent on programmes where effects are unknown. As the UK government works on strategies for meeting the ambitious Sustainable Development Goal targets of leaving no one behind by 2030, we recommend this includes a strong commitment to work on both generating and using more and better evidence to inform decisions about policies and programmes.

 

 

 

 

 

 

 


References:

The full reference list for all studies included in the systematic review is available in the summary report and full technical report:

Snilstveit, B, Stevenson, J, Menon, R, Phillips, D, Gallagher, E, Geelen, M, Jobse, H, Schmidt, T and Jimenez, E, 2016. The impact of education programmes on learning and school participation in low- and middle-income countries: a systematic review summary report, 3ie Systematic Review Summary 7. London: International Initiative for Impact Evaluation (3ie). Available at: http://www.3ieimpact.org/media/filer_public/2016/09/20/srs7-education-report.pdf

Snilstveit, B, Stevenson, J, Phillips, D, Vojtkova, M, Gallagher, E, Schmidt, T, Jobse, H, Geelen, M, Pastorello, M, and Eyers, J, 2015. Interventions for improving learning outcomes and access to education in low- and middle- income countries: a systematic review, 3ie Systematic Review 24. London: International Initiative for Impact Evaluation (3ie). Available at: http://www.3ieimpact.org/media/filer_public/2016/07/12/sr24-education-review.pdf