Written evidence from Dr Genevieve Maitland Hudson (KCI 31)

Background

I am a former employee of Kids Company. I worked there between August 2008 and December 2009. Since leaving Kids Company I have specialised in social measurement and the use of evidence in social programmes. In September 2014 I was commissioned by Newsweek to write an article about Kids Company looking at its research and evidence and raising some of the questions that had long troubled me about its model and effectiveness. I wrote my article which was scheduled for publication in December of 2014. The charity was approached for comment in December and publication was initially delayed for their response. Following a phone call from the charity in January, Newsweek decided not to go ahead with publication. I subsequently published the article myself on my work blog on 11 February 2015 and linked the article to Miles Goslett’s piece on the Joan Woolard case published by the Spectator.

Evidence

This evidence is divided into two parts. The first section outlines specific issues that concerned me regarding the charity Kids Company and its reporting of its work with vulnerable children and young people. The second section outlines how the particular failings in Kids Company’s internal systems and the external monitoring of those systems could be mitigated in the future through due diligence measures and a systemic shift in our approach to evaluation.

Section 1 – Kids Company

1. Claims of Effectiveness

As I set out in my blog post on 11 February 2015, Kids Company regularly advanced statistics in support of its work. These statistics were used in the charity’s annual accounts, in its marketing materials, and in its fundraising applications. Examples include:

• “Kids Company has been evaluated 15 times since 2000. All of these evaluations have found that Kids Company’s interventions have success rates of between 80% and 100%.”
• “96% of clients aged under 16 were helped to either return to education, or sustain themselves through our additional support.”

These claims were problematic for four reasons:

1. The figures were presented without supporting evidence. Kids Company did not publish its evaluations and was extremely selective in quoting from them. This made it impossible to check the premises of their claims and determine the validity of the research methods used to advance them.
2. The terms were vague and hard to interpret. In the example above ‘success’ means little in this context without a clear definition. Successful according to what measure? Equally ‘sustain themselves’ is ambiguous. Does this mean that some clients did not, in fact, return to mainstream education?
3. The tendency to make bold claims based on multiple separate pieces of research is flawed. It is improbable that the scope, methodologies and findings of these evaluations were compatible in the way that the charity implied. The 2001 NCB report for example, presumably included as one of the 15 evaluations cited, was a qualitative assessment of the charity based on a small number of interviews with staff and young people and contained no analysis of the effectiveness of the organisation’s interventions.
4. Finally, as part of my research I conducted an analysis of public datasets to see whether they corroborated the statistics put forward by the charity. They did not. Available data on school exclusion, offending and hospital admission did not support their claims. The public record does not show that Kids Company’s presence in the boroughs of Southwark and Lambeth between 1996-2015 had a positive impact on key indicators for its client group.

2. Use of Research

It was not only statistics that were problematic in Kids Company’s use of evidence. The charity also had a muddled approach to the use of research both to inform its model of delivery and to justify its approach.

In illustrating the problems in the charity’s presentation of evidence it may be helpful to identify the different kinds of research that are typically used by social programmes. The table below sets out these different strands.

These strands of research provide different kinds of information to programmes. It is important to differentiate between them consistently.

The first two categories of research above are inputs into a programme. They provide information which the programme’s managers can use to inform their model.

Every other category provides information on how the programme itself is running.

At each stage, with each kind of evidence, it is important to ask whether the standard of evidence is high. This would typically include questions about the process of commissioning the research (was it independent? was it published? was it peer reviewed?), and about the likely validity of its findings (how many were in the study sample? was the sample randomised? were the findings triangulated with other supporting data from secondary sources?).

Kids Company was not sufficiently clear about these distinctions and did not systematically ask these questions.

Instead, very different kinds of study were typically presented together as if they had equal value in demonstrating the effectiveness of the charity. Whether intentionally or not, this was misleading since it tended to suggest that the charity was committed to using good quality evidence to inform and assess its work when this was not in fact the case.

This second table shows how the research published in Kids Company’s 2013 Annual Accounts divides between the categories. This analysis is based on an assessment of the research where it was available through publication, and through correspondence with the researchers listed in the Kids Company table. This correspondence revealed that in some cases the research presented in the chart had not been carried out, or could not be corroborated through the contacts listed. The table below includes only those studies that I was able to account for directly through published records and/or direct contact.

This table shows clearly – even before assessing the value of any given study according to standards of evidence – that the bulk of Kids Company’s investment in its ‘scientific partnerships’ was concentrated on academic research in areas of interest to the charity but not directly related to delivery. It had an interest, and funded, research that could have informed its model, but invested much less in assessing whether that model was working.

The single outcome evaluation in the 2013 Annual Accounts – which covered research going back at least 5 years – was a Tavistock Clinic study of a small sample of children and young people (29 at baseline and 22 at endline). The full study is unpublished but a presentation available online suggests that the endline ‘success rate’ was a much more modest 50%, rather than the 80-100% claimed in the charity’s Accounts. The very small sample size, however, should discourage any further generalisation from this study.

The National Audit Office report published on 20 October noted that Kids Company had provided the LSE report, the UCL needs assessment and a further report by the Centre for Social Justice – also a qualitative needs assessment – as evidence of its impact. This shows that Kids Company did not have, or was not able to present, the kind of robust evidence that would have demonstrated the effectiveness of its programme. Instead the charity provided documents that showed need, and to a lesser extent, an outline of a working model.

These distinctions are important. Camila Batmanghelidjh and Chair of Trustees Alan Yentob have claimed that Kids Company was “endorsed by Cambridge University”. The relationship between Kids Company and Cambridge was one of grantmaker and grantee. The research conducted through that relationship was not an assessment of the validity of the charity’s model, or of that model’s operation. It does not, therefore, tell us whether the charity was doing a good job.

Questions about evidence of impact are important given the continued claims by Batmanghelidjh that financial management and governance questions notwithstanding, Kids Company provided ‘exceptional clinical and financial value’. Neither published evidence from the charity, nor public datasets appear to support this assertion.

3. Monitoring Data

There is evidence that Kids Company’s monitoring data is unreliable. The number of clients that the charity supported has been in question since the closure. The charity claimed to be supporting 36,000 children and young people but only shared 1,900 client files with Local Authorities when it shut its doors.

It was clear even before the handover of client files that the charity’s claimed numbers were unlikely to be accurate.

Before the charity’s closure and the handover of files, I carried out and published a caseload analysis of the figures claimed by the charity. This analysis used evidence from social work as a benchmark. The 2012 Community Care survey of social work showed an average caseload of 25 in adult social care and 17 in children’s services.

Based on a calculation using published staff and client numbers (assuming 400 full time key workers and 18,000 receiving intensive support) Kids Company had an average caseload of 45. That is almost three times the national average in children’s services, which the charity’s leader had repeatedly described as overstretched and unable to cope.

These kinds of caseloads would not have given the charity the capacity to manage the numbers they claimed with the model they described. Camila Batmanghelidjh claimed to be “obsessed with protecting” her workers from stress and to “always have a key worker present to support that young person”. These statements are incompatible with the aggregate figures regularly asserted.

The inflation of figures may also have affected the audits that the charity’s leaders point to as evidence that the organisation was well run. The figures provided through self-assessment to Methods Consulting Ltd, noted in the NAO report, look improbable, most particularly in the light of the low client numbers revealed by the charity’s closure. The NAO report also notes that Kids Company agreed key performance indicators with the Department for Education but did not report on them in its quarterly reports.

4. Conclusion

In summary, evidence that demonstrates the charity’s effectiveness is thin. The charity itself did not invest in providing this evidence. Instead of investing in providing evidence on the effectiveness of its own programmes, research budgets were spent externally on funding academic research that was not directly relevant to the charity’s delivery. The charity’s published monitoring data does not appear to be reliable. It did not report on indicators set by the Cabinet Office as part of its grant. Only in March 2015 was evidence of impact finally requested, and none had been provided before the charity’s closure.

Equally, it does not appear that these weaknesses were spotted by the departments that funded the charity between 2002 and 2015. Concerns were raised about financial management but questions were not asked about the lack of evidence of effectiveness until 2015. A better grasp of evidence and its use in the social sector would have raised flags sooner. Some recommendations about this are given in the section below.

Section 2 – Measurement, Benchmarking and Transparency

1. Measurement

It is clear from the NAO report that the grant monitoring in the case of Kids Company was insufficient, and that grant renewal was not in any way contingent on evidence that the model was working for those it sought to help. In particular, a requirement to evidence impact was only introduced into funding arrangements with the charity in March 2015. No social programme – public, private or non-profit – should receive grant funding on this basis.

The key question for grant-makers now is: what is the minimum, reasonable model of evidence that a programme should be expected to provide? In designing that reasonable model it is important to ensure that:

• It does not introduce false incentives
• It does not distort frontline delivery
• It is not too expensive
• It does not require skills that programmes do not have and cannot hire

Put in more positive terms, the reasonable model of evidence should: reward honesty, encourage learning and contribute to excellent frontline delivery. This is not currently the case, nor was it the regime under which Kids Company operated.

What are the key parts of this model? This is a question that I have addressed in collaboration with New Philanthropy Capital and the Centre for Youth Impact in a recent publication, and on which we continue to work.

An outcomes framework alone is insufficient. The outcomes framework referred to by Richard Heaton, former permanent secretary to the Cabinet Office, in his evidence to the Public Accounts Committee on 2 November 2015 would in all likelihood not have been fit for purpose unsupported by further work on the model developed by Kids Company, and buttressed by fidelity and quality monitoring.

A social programme evidence and learning cycle should look something like this:

Without going into unnecessary detail here, a short description of each part of the cycle is given in the table below:

This model sits within a broader social programme ecosystem. Again, without going into unnecessary detail, it is important to note that (at least) two parts of this model are best delivered from outside the individual social programme itself.

Evidence to inform design will usually be found, and should usually be found, in research produced by academic departments and through organisations uniquely developed for this work, such as the What Works Centres. Similarly, outcomes measures should increasingly be understood as the proper domain of centralised organisations able to make datasets available to social programmes. This has already been piloted through the Ministry of Justice Datalab.

Social programmes themselves are generally poorly placed to produce sufficiently robust outcomes measures. If social programmes were to produce valid outcomes research, this would require investment that would take resources away from frontline work. It would also, in most cases, duplicate data collection that is already happening elsewhere, which is wasteful. Outcomes research is also best conducted at intervals. Many programmes will only need to evidence outcomes every 2 to 3 years. Internal monitoring of fidelity, quality and capacity will be infinitely more important for regular programme improvement.

Incentivising and investing in better sharing of data across social programme delivery would be a more efficient approach to the rigorous use of outcomes data. Further investment and encouragement by government could be usefully directed at growing this approach to outcomes measurement.

It is also best for quality monitoring – inspection – to be carried out by independent agencies. Kids Company’s refusal to register with OFSTED or the CQC meant that important external oversight that would have raised concerns sooner was also evaded. This absence of quality inspection seems also to have been missed in government grantmaking. Cross-checking against a tool such as the cycle given above would have made such an oversight less likely.

2. Benchmarking and sharing measurement             

There is an equally important need to incentivise and invest in the development of shared measurement frameworks within the social sector. One of the greatest difficulties in assessing Kids Company’s value for money has been the absence of shared indicators that would allow for a reasonable comparison of its work with that of other providers. The use of aggregate ‘reach’ numbers as proxies of effectiveness is particularly unhelpful. It encourages inaccurate reporting and gives very little insight into programme capacity.

Benchmarking of, for instance, caseload in 1:1 support programmes, offers a reasonable starting point for comparing usefully across relational support programmes. This would be an equally valuable measure for many central and local government programmes. Given the unreliability of the current measures, this would be an approach that could usefully be piloted in the Troubled Families reporting across England and Wales.

Incentivising the take up of standardised measurement tools of all kinds is crucial here. We won’t be able to compare results in any useful way until we have some element of standard measurement in place.

It is important not to see benchmarking as a competitive practice aimed squarely at bringing down costs and increasing and intensifying workloads. Used in combination with robust outcomes measures and beneficiary feedback it offers the potential to free up the debate around appropriate support and demonstrate the importance of reasonable workloads for achieving long-lasting positive change for beneficiary groups.

3. Transparency

The experience of Kids Company demonstrates the vital importance of open and transparent publication of the evidence of effectiveness of social programmes, both when that evidence is positive, and when it is negative. A culture of sharing evidence, and learning from it should be the foundation of good practice and should be actively encouraged and rewarded. This needs to happen at every level, with funders and commissioners publishing transparent data as well as providers. There is considerable progress still to be made in this area.

November 2015