Education Committee
Oral evidence: Accountability Hearings, HC 341
Tuesday 19 December 2017
Ordered by the House of Commons to be published on 19 December 2017.
Members present: Robert Halfon (Chair); Michelle Donelan; James Frith; Emma Hardy; Trudy Harrison; Ian Mearns; Lucy Powell; Thelma Walker; Mr William Wragg.
Questions 303 - 407
Witnesses
I: Sally Collier, Chief Regulator, Ofqual, and Roger Taylor, Chair, Ofqual.
Witnesses: Sally Collier and Roger Taylor.
Q303 Chair: Everyone, welcome. Thank you very much for coming today. Just for the benefit of the tape and the audience outside watching it on the internet, please could you give your names and positions?
Roger Taylor: I am Roger Taylor, the chair of Ofqual.
Sally Collier: I am Sally Collier, the chief regulator at Ofqual.
Q304 Chair: Thank you, and could I ask you both to speak quite loudly to the microphone? We have quite a lot to get through, so the more concise your answers, the better.
Your report “Perceptions of AS/A-levels, GCSEs and other qualifications in England”, published in April 2017, stated: “Despite the proportion rising, only 47% of stakeholders thought that AS/A-level standards are maintained year on year, and only 41% thought that the marking of AS/A-levels is accurate.” Would you agree that it is incredibly worrying, and what do you think should be done about this?
Sally Collier: Yes. If you look at the absolute numbers of those percentages, it is everybody’s job and our job in the system to continue to drive those up. I would say that they have been like that for a number of years; this is not a recent phenomenon. We were quite pleased that in the last survey we saw increases in those perceptions. They have a range of questions, sometimes 10 percentage points or more.
The reality of what is undertaken in the system and the validity of those qualifications that we are confident about versus the perception—we need to continue to get the messages out there about what is actually happening in terms of validity. I don’t know whether Roger wants to comment.
Roger Taylor: I would agree with that, and I would point to our research showing that the quality of marking in this country does compare favourably with the quality of marking around the world, and that our job is to ensure that the quality of marking is the highest it possibly can be. The results of our work in that space are bearing fruit in the rising levels of confidence.
Q305 Chair: If I could gently challenge you on that, you mentioned that this has gone on for some time. It is still nearly 50% of stakeholders, in terms of AS or A-levels standards being maintained. Just because something has always happened does not mean it should continue to happen, and even if it has only improved slightly, surely more should be done to dramatically reduce that figure.
Sally Collier: To dramatically increase that figure, yes.
Chair: Or reduce the number. It is nearly 50%.
Sally Collier: I come back to my point about what is actually happening in the maintenance of standards. Ofqual’s core mission is to maintain standards. We have done some excellent work in that area. This is the perception. Yes, of course we need to continually work on the perception. I think we are a pretty transparent organisation, with regard to getting out there what has actually happened in standards of qualifications. Yes, we always need to do more.
Q306 Thelma Walker: Sally, you started in the role at a time of an incredible amount of change, and in your inaugural speech as chief regulator you said, “I want to continue for my stint here to get out and about and listen to the people at the coalface”. Have you done that, and how?
Sally Collier: Yes, indeed. That has been one of the most rewarding aspects of the job over the last year. I think I have had 23 visits to schools, colleges and training providers—it was really intense in the first period—listening to students, listening to teachers, listening to exam officers, talking about how qualifications have been delivered in the schools and colleges, what teachers feel about how the new reforms have been landing. That has been a hugely important part of the job, and not just in schools and colleges but with all the teaching unions and teacher associations. Hopefully, they would say we have developed good relationships and we are constantly listening.
Q307 Thelma Walker: One of Ofqual’s key statutory duties is to promote public confidence in the assessment system, but what if I were to tell you that the survey has shown that only 2% of the general public are aware that the national reference test has been introduced at all?
Sally Collier: Yes. I am not hugely surprised. First of all, it is new. We ran the first proper test this year. It will not be used in earnest until we are confident that the data can be used. I am not surprised at that level of confidence. As we increasingly go through the cycles, we will ramp up communication about those tests, but they are not tested in the same way that other types of examination are used. Pupils do not get their individual results.
Q308 Thelma Walker: If we also say that only 50% of teachers themselves know about it, that I find a little worrying.
Sally Collier: I come back to my response to the Chairman earlier: we can always do more to raise awareness, but not every school, of course, is affected by the national reference test in any given year; there are only 300 schools in any given year where the pupils take the one-hour tests.
Q309 Thelma Walker: You have mentioned ramping up communication. What, in effect, does that mean?
Sally Collier: It might mean a series of things as we prepare for potentially using the data. I say “potentially” because we need to be confident that the data is robust and that we can start to use it in the awarding process. Over the last year we have used a range of communication methods. We have used videos, we have used blogging, and we have used being out and about on conference platforms. We would use a range of those methods.
Q310 Thelma Walker: Would you say that timescales are important with this, though? If you are talking about communicating or initiating something like this new national reference test, do you think the timescales are correct in that professionals have that time to get on board with something, to have an understanding and to be able to implement it?
Sally Collier: We ran a pilot—a pre-test—of the national reference test. We ran the first live running this year. We have had no problems with schools understanding. We have communicated very directly with those schools that are going to participate, and we have had no issues with the running of that test in a small number of schools.
Q311 Mr William Wragg: Good morning, both of you. Sally, you will remember your pre-appointment hearing, which I am sure is now engraved on your mind, when I asked a very—
Sally Collier: I remember your questions, William.
Mr William Wragg: It was a good job, too, because, funnily enough, I am going to ask you the same question again—I think the best examinations do that, do they not? It is maybe in a slightly different way. I asked if you had ever said, “No, Minister” in your previous role, so in this role have you yet said, “No, Minister,” and could you give an example?
Sally Collier: In all of our work over the last year you will have seen that Ofqual has been very transparent about what we think works well in the system and where we think there are risks in the system. We have done that across a range of subjects, in our publications. We have had 17 public consultations. We have published 20 research reports. We have a very public transparency role. Of course, we have discussions with the DfE and with Ministers—
Q312 Mr William Wragg: Is there a specific example that stands out to you with a Minister—and do not worry about embarrassing them—where you have said, “No” or perhaps, “Amend your plans, Minister”?
Sally Collier: We have said this publicly and privately in the forthcoming reform of vocational qualifications, where we have commented on the arrangements for regulation and the speed of implementation. Indeed, in reflecting on national assessments and primary assessments, we have pointed out where the STA can work on particular issues or we have pointed out risks in the system, like the forthcoming new tests in primary assessments. There are a range of occasions over the last year where we have done that.
Q313 Mr William Wragg: Just further to that, on perceptions of independence of Ofqual as the regulator, would you be happy to reinstate the question in the next survey as to whether respondents feel that Ofqual is, “Too close to Government”? I believe it was removed after 2015.
Sally Collier: Yes, and in my preparation we looked at that as well, and I have asked why it was removed. I think it was in there originally because Ofqual was new. We have had 15 of these perception surveys. I think it was in there originally because nobody really understood who we were or what we did. We certainly reflect on whether it should be reinstated, but for me it is not about Ofqual; it is about the outcomes for students. If we are doing our job properly, that independence should come through because it will be transparent in our work.
Q314 Mr William Wragg: Of course, but if a question that asked directly about independence is removed or dropped off, for whatever reason, is there a risk that perceptions of a lack of independence might creep back in?
Sally Collier: Potentially, but we hear immediately in today’s social media world whether there are any fears or comments about our independence. Regardless of whether that question is in or not, it is pretty immediate whether there are concerns about our independence.
Mr William Wragg: Thank you.
Chair: We are going to move on to T-levels now.
Q315 James Frith: Good morning. There is a history of Government attempts, of both colours, to engage with employers in the development of qualifications: the Tomlinson review, the Leitch review of skills, and more recently the diploma. How do we ensure that employers are not just fatigued by these attempts to have vocational qualifications valued, and, from your point of view, what are the risks to the successful development of T-levels?
Sally Collier: Perhaps I will take the latter, but maybe Roger can take the former.
Roger Taylor: Sure. I think that one of the things that fatigues employers is the number of attempts that have been made at this. Where we are now, we need to have a wholesale programme of the one that is being put forward to address this issue.
In terms of getting employers’ support, the key issue is that this is a process that will take some years. The final T-levels will not be out until 2024. The most important thing is that there is clarity that when in five or 10 years’ time somebody thinks they can see some way of improving this, we do not move back into another process of wholesale reform. It is clear that we are set on a path here and that we are going to deliver these effectively for employers. That commitment and the existence of an independent regulator to ensure standards are maintained will be one part of that, but crucially the IFA plays the central role in terms of employer engagement and making sure that they are getting what they need, and the whole system has a role to play. It is being clear that we are in this for the long term and we are going to deliver qualifications and education systems that are fit for the purpose of giving young people the skills they need.
Q316 James Frith: To that point, do you think it is helpful to talk in terms of being one or another pathway—academic or vocational—when employers of all sectors and types value academic rigour as well as vocational? Do you think that risks undermining from the start any kind of aspirational requirement for a qualification to be valued, protected, invested in and pursued?
Roger Taylor: To my mind, the key thing about the value of the qualification will be the degree to which it opens up opportunities. I would caution against having an answer of, “One size fits all.” There are some career routes where employers will want perhaps to employ people at 18, but also a significant number of people will be employed as graduates, and T-levels in some circumstances may need to support people into university to complete their training before they go into it all, but for another career progression that may not be how the employers work. We need to caution against saying, “This is the right way for them all to work.”
Sally Collier: On Ofqual’s role in the implementation of T-levels, we are the independent regulator for England, and we anticipate that, regulating the qualifications within T-levels, they are of course broader than just the qualification with the work placement. That is not a matter for us, but we would expect to regulate them. I feel strongly that they should be, and we will be working very closely with the IFA to ensure that our respective duties and powers are clear and that we can work alongside them to regulate them.
Q317 Chair: What is your view about the fact that there is only one awarding organisation for the different T-level qualifications? Are there not significant potential problems with it?
Sally Collier: To some extent we are agnostic about whether there is one or many players in the market. The Government have made a choice about the shape of the market. It is our job to make sure that, whoever provides those qualifications, they are valid qualifications for learners and that we do not repeat the cycles of what has happened previously.
Q318 Chair: Do you not think there should be one in reserve in case something goes wrong?
Sally Collier: The Government are well aware of looking at all of the risks surrounding a single provider. It brings benefit in consistency, in clarity, in reducing confusion about qualifications, but of course there are risks, risks of incumbency. All of those risks need to be carefully planned for, and we have a statutory duty relating to the efficiency of the market, so we will obviously be working with the DfE about analysing those risks and making sure they do not come to pass.
Q319 Chair: I am trying to understand whether you think the risk is greater than the rewards of just having one award organisation.
Roger Taylor: I think you have identified a potential risk, which is there only being one provider available to deliver something. I am not sure whether having a specified reserve provider is necessarily the right move, but what you are definitely right about is that one implication of going down this route is that we do need to ensure that the market continues to have a sufficient diversity of qualification suppliers so that the commissioners of those qualifications do not find themselves in a situation where they are not able to select from suitable candidates.
Q320 Chair: Finally, the roll-out of T-levels has been delayed, and it will be a pretty long time before they all come to fruition. Given our skills deficit, there are a lot of questions being asked about that. Do you have a view?
Roger Taylor: I would say that because of the skills deficit we need to adopt an ambitious timetable. I would say we have adopted an ambitious timetable.
Q321 Chair: Even with the delay, do you still think it is an ambitious timetable?
Roger Taylor: Yes.
Sally Collier: Yes, and it is hugely important for all those students who will be collecting their T-levels. The most important thing from our point of view is the quality, valid, independently assessed product, where we do not see what has happened before in the cycles. That is our priority.
Q322 Trudy Harrison: One of the risks that you have not identified is the progress in terms of the placements. Around three months would be spent with an employer in industry. Why would an employer sign up to that when they could have a level 2 apprentice?
Sally Collier: I am afraid the work placements are not a matter for us. We are regulating the qualification, so I am sorry but I cannot answer that.
Q323 Trudy Harrison: Do you see this as a concern and a risk to delivery, though; the ability to secure those three-month placements? Also, moving on, I understand that a T-level means that you would not be able to then join a level 2 apprenticeship.
Roger Taylor: Just to be clear, we do not have a reason to believe at the moment that the three-month work placement undermines the validity of the qualifications within a T-level, but risks on that are a risk to validity. I am saying that cautiously because people may raise issues with regard to that, but that is our stance at the moment. It is not properly our arena to look at that.
Q324 Michelle Donelan: I just want to clarify something. Would you say that preparing for T-levels is a major endeavour of your work and a major component of your work at the moment?
Sally Collier: Yes. I think I said at my appointment hearing, and in subsequent speeches, that what I wanted to do during my tenure here is really beef up Ofqual’s role and our resources and our focus on the vocational education. Yes, we have transformed the way we work to support both apprenticeships and T-levels and all of the other qualifications that we regulate in the vocational space.
Q325 Michelle Donelan: Do you predict that in the future this will become a major component, and Ofqual will in effect be changing, so the very shape of it will be changing and dynamic?
Sally Collier: We certainly changed the way Ofqual works to support both general and vocational qualifications.
Q326 Michelle Donelan: On that note, our predecessor Committee said earlier this year that it recommended that the role for Ofqual be enlarged in the apprenticeship arena and its regulation of that. Do you subscribe to that view and believe that is the direction of travel you should go in?
Sally Collier: If Government were to recommend that, we would accept that plan for its implementation and of course get on with it. That is not the case at the moment. We are the external quality assurer—one of a number—for apprenticeships. What we have been doing during the last 12 months is getting on with making sure that they are quality end-point assessments in those standards. We have looked at 100. We have agreed to be the external quality assurer for about half of those, and we have 50 in the pipeline. We have been getting on with it. We are willing and ready to do more. The other types of quality assurer in the system also bring skills and experience and value. Whatever happens, we need to work closely with industry bodies and professional associations that are also in this space.
Q327 Michelle Donelan: Just to clarify, I know that you might be willing and ready to take on that extra responsibility, should it come, but do you believe that it should be in the hands of Ofqual? With your unique position and the experience that you have had to date, do you believe that it would be best placed?
Sally Collier: It is difficult for me to say what is happening. I do not have a full picture of what is happening with the other quality assurers. It is for others to assess the balance of who is doing what in this space and who is bringing good practice. It is not for me to say it should be us. All I know is that we are getting on with the job as we are asked to.
Q328 James Frith: I will move on to apprenticeships. The Government are big fans of talking about parity of esteem being sighted, and I do not think it is demonstrated at all if you consider the mess that we are looking at with apprenticeships. Sally, you yourself have said previously that it is not a qualification. We now move towards T-levels, where apprenticeship will be a form of qualification. It is not included within the destination measures for a school, and so there is a disincentive for schools to be putting young people on a path that is suitable to them. We now have statutory guidance to be issued that will try to counter that. We have a drop of 63% in apprenticeships in the first year of an apprenticeship levy. We have a degree apprenticeship model that is funded by the tuition fees of traditional degrees but is not recognised as higher education. It is an absolute mess, is it not?
Sally Collier: What I can talk about is what we are doing for the bit that we are responsible for, which is to be the external quality assurer for apprenticeships. We have taken into our regulation the end-point assessment as a qualification type so that we can regulate against it. I do not know whether Roger wants to make a final comment.
Roger Taylor: I would.
Q329 James Frith: If I am setting out to offer a quality assurance model in any walk of life, I will have some clarity as to the counterproductive or cross-pollinating or counterintuitive facts, as I have outlined there. That is just scratching at the surface. Is there not a moment where, as the regulator, you ask the Government to get their house in order on something that they proclaim but do not demonstrate as being valued at the same level as A-levels, HE, pursuit of degrees and so on?
Roger Taylor: The key issue about the value of new apprenticeships will be the degree to which they deliver for young people the skills they need to get the job and have a successful career. That is the overriding concern. Ofqual is deeply engaged in that process and working with 100 apprenticeships to ensure that happens. The overarching responsibility to make sure across the board that it works lies with the IFA.
There are pros and cons to the mixed model of current regulation. The con is that it makes the IFA’s job harder in ensuring the consistency—that all of these are delivering the quality of qualification. It does not make it impossible; it just makes it harder. The pro is that you get a much wider range of organisations involved in that quality assurance. Right now, we do not have reason to believe that the issues you are addressing will necessarily undermine—
James Frith: With respect, for every 10 employers engaged—this does not carry across—six are no longer engaged this year, because there has been a 62% drop, apparently. You cannot uniformly apply that, but nevertheless—
Chair: That is just over a quarter; it is not over a year.
Q330 James Frith: We do not expect it to be dramatically different. I challenge this idea that having an increased number of providers is automatically a good thing, when there is such a lack of clarity on a pathway for a young person to take an apprenticeship. I wonder at what point do you, as the quality assurance regulator, say that the quality is not up to scratch.
Sally Collier: Can I come back to you? The body with responsibility for the quality of the entire apprenticeship system is the IFA. That is what it was set up to do. Whether the whole system is working, and whether there is clarity about the number of placements, is the responsibility of the IFA.
We are currently undertaking a specific role as the quality assurance provider for a proportion of those people who want to be quality assured by us. If there were any serious risks to the validity of those end-point assessments, then of course we would point that out. These are new, and we have been working very hard with those 100 providers to make sure that there is a quality end-point assessment at the end of it.
Q331 James Frith: Just a final, very quick question. End-point assessment: that is a fair point. I came to Parliament wanting to share news of completion rates being pretty poor for apprenticeships, and it turns out that start rates are too. That end point that you talk about is not in isolation of the journey that a young person has to take. My point is that surely you should be reaching into the system and ensuring that these things are lined up, because the description I have offered is anything but straightforward.
Sally Collier: We are reaching into the system. We are also part of—and the IFA has set up—a quality alliance with ourselves, with Ofsted and with the higher education quality assurers, so that we are talking about how the system is performing as a whole.
Q332 Chair: Can I move on to the end-point assessments? There is a view that it is a bit like a Ben Hur movie with a cast of thousands, in terms of the assessors of the assessors and the different organisations that do it—the IFA, yourselves and the trade bodies. Is it your view that there are too many characters assessing the qualifications? Do you think it should just be you? Do you think it should just be the IFA? Are you happy with the mixture?
Sally Collier: I have been on the record as saying that it is confusing, but a lot of work has been done over the last 12 months to look at the respective roles—the IFA and ourselves have very different roles. It would be wrong of me to sit here today and say, “It must be us.” We need to get further clarity. We will get that as we go along. Whether it should be us or anybody else, what matters is that the students coming out with those end-point assessments need to be assured that there are the same standards—
Q333 Chair: If it is not you but a trade body doing the end-point assessments, how do you maintain the quality if you are not involved at all?
Sally Collier: We are involved. I think—others will be the judges—that we have done a good job in the ones that we have—
Q334 Chair: You are doing 40 standards. Is that right?
Sally Collier: We have looked at 100. We expect to get more. We have approved 40, and there are another 48 in the pipeline.
Q335 Chair: If the end-point assessment is done by the trade body, not by you, how do you maintain the quality?
Roger Taylor: Just to be clear, we do not maintain the quality of those that we do not directly regulate. We have offered to work in partnership and we are working very successfully in partnership with other organisations in this space, enabling them to draw on our regulatory powers. Were our role to be expanded, it is important to be clear that it would be essential that we continue to draw on the expertise of the many trade bodies that have enormous strengths that they can bring to this space. Where we are not currently the EQA, or working in partnership with the EQA, the responsibility for ensuring that the quality is consistent lies with the IFA.
Q336 Trudy Harrison: I have a follow-up question about the T-levels. You were slightly dismissive, I thought, about the three-month work placement. How are you involved in assuring quality of that three-month work placement?
Sally Collier: Sorry. I absolutely do not intend to be dismissive. We are not involved in assuring the quality of the three-month work placement. It is not a qualification, so we are not involved in that.
Q337 Trudy Harrison: Surely the three-month placement is an integral part of the T-level. Should you not be involved?
Sally Collier: It is an integral part of the T-level, but it is not a qualification. Where we are involved is in the T-level qualification.
Q338 Chair: Just to go back to my previous question, are you saying that you do think there were too many players or not, in terms of the end-point assessment of the assessors?
Sally Collier: It is too early to say. There certainly needs to be clarity about who is doing what.
Q339 Chair: We are going to move on to GCSE and A-level qualifications. Before I come to that, I am just going to ask one question. If I could ask you about the 52% increase in GCSE grade changes after re-marks and the fivefold increase in changes by two or more grades, I think Schools Week has worked out that that is equivalent to 2,500 children going from D to B grade, which may not sound a lot, but it is equivalent to every year 11 and 12 school shifting from a fail grade to a good one. What is your view about that?
Sally Collier: We are very disappointed that these grade changes have happened, and we have had an intensive period of evaluation as to why this has happened over the last few weeks, as you would imagine, since the reviews have been completed. We are currently in discussion with exam boards about what should happen.
Q340 Chair: A D to a B is dramatic; it is not just a D to a C.
Sally Collier: Exactly, yes.
Q341 Chair: Does it suggest that something endemic is wrong in our marking system?
Sally Collier: There are a number of things here. Our evidence has shown that there was not a problem with the original quality of marking. We have had a look at marking consistency metrics across all the exam boards, and the level of consistency in that marking was very similar to last year’s. There was nothing that happened in the original quality of marking that suggests that there was a problem.
There was a problem and there has been a problem with the reviewing. You asked whether it was endemic. That problem is not consistent across the exam boards. What we have seen is that some exam boards have implemented it very well, and some have not.
Q342 Chair: What are you doing about it?
Sally Collier: We are spending a lot of time looking at what needs to be done for immediate re-marking of the November series resits that happened in November. I would not be able to talk about the specific regulatory action that we are taking with those exam boards that did not follow our rules, but we will ensure that it does not happen again in subsequent series.
Q343 Lucy Powell: Just related to that, obviously there is a lot of change coming in with GCSEs and A-levels at the moment. Do you feel it is sufficient, and what are you doing to address the fact that still only 32% of employers are aware of the new 1 to 9 system, and in fact teacher awareness of the new grading system has decreased over wave perceptions 14 and 15? What more can you do, and are you happy with those figures?
Sally Collier: Yes, obviously a very extensive, intensive reform programme happened over many years. I believe in 2011 the first decisions were made about the new qualifications. In terms of 9 to 1 and awareness, we ran an extensive campaign. We had video materials sent to all schools. Ten million views of the videos, tens of millions of hits on social media—I have the numbers in front of me. We ran our own tracking of particular groups of parents and students, and there were quite significant improvements in awareness and understanding over that period.
You mentioned employers. Yes, their awareness and understanding is lower—I think 5.5 million employers. Some of these awareness figures are higher than for the existing A* to G system. It will take some time for every employer in the country to understand 9 to 1—
Q344 Lucy Powell: What more are you doing?
Sally Collier: In fact, we have just this week—it is nothing to do with this Committee hearing—released an updated video on 9 to 1. We will run a further campaign for next summer and in subsequent summers, and we will be doing some very specific targeting of particular groups of employers. I think we have very good levels of awareness, as you would expect, in schools and with head teachers and students. We will look at some specific targeting of parents and employers for the next wave.
Q345 Lucy Powell: As a parent myself, I have to say that I think parents are deeply unaware of what is coming. In that regard, what are you actually saying about what a level 4 and a level 5 are? When I met your predecessor about 18 months ago, at that point Ofqual was very clear that the bottom of a level 5 was the equivalent of a bottom of a level C. That has changed over the last 18—
Roger Taylor: The bottom of level 4 is equivalent to level C.
Lucy Powell: No, it is now.
Sally Collier: No. We have always been clear, in terms of our pegging to the previous grades, about level 4 and level C. That is what we use in our predictions—
Q346 Lucy Powell: Honestly, when I met with Amanda Spielman 18 months ago, at that stage the bottom of level 5 was the bottom of level C, and it has changed quite a bit over that time.
Sally Collier: What have changed are the accountability measures to do with what is a good pass, what is a standard pass. That is not a matter for us; it is a matter for the Department. How these grades are used, and how the accountability measures work, is a matter for the Department, but we have always been clear that we have pegged—
Q347 Lucy Powell: Do you think this is part of the problem for people to understand what is the equivalent of a pass, because we now have a lot of grey area between a 4 and a 5 and what really we are benchmarking against?
Roger Taylor: It is important to distinguish between a pass in terms of accountability measures and a pass in the real world, because Ofqual does not define a particular grade as a pass. People set their requirements for a job application or for a place in higher education, and they vary. What is a pass to go and study a subject at a particular university bears little or no relation to what the DfE regards in its accountability metrics as a pass. What we want to ensure is that whatever the grade is, it is an accurate—
Lucy Powell: Do you not think that is part of the problem, though? What is it you are aiming for? You are Ofqual and you cannot answer that.
Q348 Ian Mearns: Just for clarity, when you are talking about an accountability measure, are you talking about things that would count for a league table, for instance?
Sally Collier: Yes.
Roger Taylor: You are absolutely right. It is important that young people understand what they are aiming for, but that will very much depend on where they are trying to get to in life and what their ambitions are.
Q349 Lucy Powell: This probably is why there is so much confusion.
On a related matter, were you aware—I am sure you are aware—that in the maths higher paper of this GCSE that people got their results for last summer, the pass mark was 18 out of 100? Do you think that is a good experience for students?
Sally Collier: Perhaps I could talk about how that came about. The higher paper is targeted obviously at the higher grades. We are not surprised by that figure because the exam boards targeted about a sixth of the questions at each grade, so 4 and 5. So we would have expected 18%, given that there were no questions at the lower levels. We expect to see similar in future years.
Q350 Lucy Powell: What was the level in that paper for somebody getting an 8? Statistically speaking, as somebody who did maths A-levels and all of that, if the pass was 18, then I would expect a level 8 to be only about 45%.
Sally Collier: That is not surprising, given the nature of that paper.
Roger Taylor: Just to be clear, the intention is that—I do not have the figures—for a paper to work effectively, you want the grade boundaries spread as evenly as possible across the full marks collected by the paper.
Q351 Lucy Powell: No, I understand that. I totally understand how you statistically do papers. I cannot understand, if I compare with previous years, the experience of a student sitting a paper for which four-fifths of that paper is something that they cannot even do. These are students who would be wanting to go on to do maths A-level.
Roger Taylor: It is worth pointing out that it is true of both the higher and the lower-tier papers that for both of them there will be a cohort of candidates who struggle with the paper and get lower marks. It is the nature of all exam papers that that will occur.
Q352 Lucy Powell: Yes. I am not talking about that; I am talking about the pass mark.
Roger Taylor: I believe it is right that the majority of grade 4s were awarded on the lower-tier paper.
Q353 Lucy Powell: Are you happy with the pass mark for the paper? Do you think that is a good thing?
Sally Collier: Yes, we are happy. I take your point. There is statistical methodology on whether the exam performed as we expected it to perform, and you are talking about the student experience. The other thing that we are doing is that we have just had 10 teacher workshops across all of the newly reformed subjects this year, talking about the student experiences, about what worked well and what did not work well. Of course we will reflect that, but the student experience is coming back into what—
Lucy Powell: I think you should. If we want students to go on and be encouraged to do A-levels and so on, especially in key subjects such as maths, to sit there and feel that a paper that they feel very well prepared for is really almost impossible in aspects, I do not think would encourage them to go on to do A-level, and I say that from personal experience.
Q354 Chair: Do you think that this system is better, or worse and more confusing?
Sally Collier: As you know, I do not come from this sector, but what I can say after 18 months of being immersed in it is that the validity of these qualifications, the robustness of these qualifications and the way that the new qualifications have been put together is better than those that they have replaced.
Q355 Lucy Powell: On what measures?
Sally Collier: On the quality of the assessment. There were all sorts of problems previously with the way that qualifications were made up, with the coursework. In terms of how these qualifications have performed as assessment instruments, we have been very pleased with the new qualifications over the summer.
Q356 Emma Hardy: By fixing the number of children who are able to obtain the different grades, do you agree that this rations progress by concentrating on what the learners can do or do not know in comparison with their peers, rather than acknowledging the achievements that they have made?
Roger Taylor: It is important to understand we do not fix the grades statistically. We think it is a really important piece of evidence in deciding where grade boundaries should be, because in order to assess the difficulty of a test, one of the most crucial bits of information is how young people perform when they take that test. It is a starting piece of information in determining where grade boundaries should be set. In the comparable outcomes framework, that is then assessed against the actual work done and a view come to as to where the grade boundaries should sit, based on the actual quality of work.
I think the point you might be referring to is that, with the introduction of the new exams, we have laid greater stress on the statistical methods in order to protect pupils. Otherwise, the first cohort of pupils taking a particular exam will underperform just because it is new, and you get a “saw tooth” effect, where you get a drop in performance simply because it is new and people are less familiar with it. We felt it was appropriate to protect young people against being adversely affected simply because new qualifications were being brought in.
Q357 Emma Hardy: You could have children next year doing the same tests as they have done this year and obtaining a lower grade just because of the movement around the grade boundaries, and that does not seem fair to me. The implication is that as a school you can only improve if another school underperforms because—
Sally Collier: Can I take that? At an individual school level, an individual school can do better or worse in any given year. That is absolutely not the case that there is a fix at school level. In any given year—
Emma Hardy: There is a national fix.
Sally Collier: I would not describe it as a fix; I would describe it as an approach that has been tried and tested for many years by many academics and statisticians. In other countries, the use of statistics in awarding is very common. In any given year, it is the balance between the use of those statistics and the use of other sources of evidence. At school level, schools go up and down—in some cases quite significantly—in any given year. That happens. We call that school variability. We predicted that this year there might be more variability given the new qualifications; there was not. Schools have responded incredibly well to the new qualifications. That is at school level.
At the national level, yes, we use a framework using the statistics, because otherwise research would suggest it is almost impossible to predict the standard in new qualifications. The chief examiners look at real work and real scripts against the predictions and say, “Does this look right? Should this be any different?” In any given year they can come to us and say, “We can’t meet these predictions. We would like to report an exception.” I think we had around 20 exceptions this year. Exam boards do bring to us that additional evidence.
Of course, in future years—we talked about the national reference test—we will be bringing yet more additional sources of evidence into the pot so that we can absolutely reflect where performance is improving at a national level or decreasing.
Q358 Emma Hardy: There are two other things that are really concerning me at the moment, particularly with regard to the impact they are having on children from more deprived areas. One of them is that the cost of re-marking exam papers is having to be met by the school, and parents cannot afford for those costs to re-mark and schools cannot afford to pay for the cost of re-marks. As we mentioned earlier, the number of re-marks that are having to happen means that you are only going to really get a re-mark if you go to a school with lots of money, like an independent school, or your parents can afford it.
The other issue that is really concerning me—and again it is hitting children from deprived areas—is the impact of decoupling AS-levels and A-levels. The Sutton Trust has just put a report out saying that children from deprived backgrounds have lower predicted grades, which has an impact on where they can go to university, because of that decoupling. I wonder what your thoughts are.
Sally Collier: Shall I take the first and Roger can take the second? On the subject of re-marking and the cost of re-marking, it is absolutely our job to make sure that the system is as fair as possible for all students and all schools. Yes, there is a cost, and we had a debate—in fact, I think you had a debate at the previous hearing—about whether there should be a cost. If there was not a cost, or if there was a lower cost, everybody would be putting in for re-marking, and we would simply be re-marking everything again. We have tried to say that there should only be a change in mark or in grade if there is an error. I know that perhaps to the general public out there that is stating the obvious—of course there should only be a change when there is an error—but that is not what has been happening over many years.
Q359 Chair: How do you know there is an error if it is not re-marked, and how do you solve the problem of very low-income students not being able to afford a re-mark?
Sally Collier: Of the school not having the funds to pay for it?
Chair: Yes.
Roger Taylor: Our first approach has been to ensure that you do not get an advantage through being able to buy a re-mark because it will only change things where there is an error, and you do not pay, obviously, if there actually is an error, so people would not be penalised. If we get that bit right, I think it will go a long way to addressing the problem. It does not wholly address the issue of whether fees should be paid for in a different manner than currently paid for. Most of them are paid for by schools. Again, that is more a policy decision, because if it were reduced to zero we would effectively end up with a double-marking system that would put a significant cost on the education system.
Q360 Michelle Donelan: What do you mean by, “It has only altered the grade if there is an error”? Surely that is stating the obvious. It must have always been that case.
Sally Collier: No.
Q361 Michelle Donelan: If it had not been marked correctly and then it was graded differently, that is in effect an error, is it not?
Sally Collier: That is an error, but what has happened and why we changed our rules last year is that marks were being changed simply because a professional had a different—“You gave an essay 16; I gave an essay 17.” There was nothing wrong with the 16. That is the inherent nature of the subjectivity of some of them. It only happens in particular subjects—it happens far less in physics and maths—such as those subjective subjects where we want extended answers and have longer-mark questions. We could get much more accurate marking by having multiple-choice questions.
Q362 Michelle Donelan: If Thelma marked it and gave it a 16, as you said, and then I marked it and gave it a 17, that means I will have been marking that cohort at a higher grade than Thelma, so you did not have accurate marking then. I appreciate that there is a degree of subjectivity, but if we are saying that one person can mark at a higher grade than another—
Sally Collier: We are not. That is what has been happening in prior years. Now our rules for the exam board say that you have to change a mark only where there is a misapplication of the mark scheme or where clearly the original marker’s judgment has somehow gone awry and there is a mistake. You should only change marks on that basis.
Roger Taylor: I would also point you to the research we published in the summer, which evidenced that there was a clear bias that, when asked to re-mark, people just tended to be generous. There was a clear bias towards seeking out ways to be kind. That obviously was working in a way that was unfair to all candidates.
Q363 Emma Hardy: Yes, which, as we know, has impacted on children from deprived backgrounds.
Just going back to the second question I asked you, on the decoupling of the AS and the A-levels, as the Sutton Trust has shown, children from deprived backgrounds have lower predicted grades. This is impacting on which universities they get chosen for. What do you think about that?
Roger Taylor: I would make two points. First, on the lower predicted grades, it is also true that people from more deprived backgrounds performed less well on AS-levels, and there is no reason why predicted grades should be worse than AS-level performance in that regard.
Q364 Emma Hardy: The Sutton Trust report says they are.
Roger Taylor: I appreciate that. It is an issue of concern. Just to reflect on the decoupling of AS and A-levels, the decision to do that was based on a view that both approaches are equally valid in terms of the qualification that was delivered. While one was plainly in line with the intentions of Government policy and the steer from the DfE, the other was not. That is the rationale for moving in that direction, with a desire to move towards more summative assessments to test people’s ability to command a broader range of knowledge.
The issue you identify around unfairness of predicted grades is interesting, but it is not one that is necessarily best addressed by changing the qualification system or doing anything that would undermine the validity of the qualification system. I would address it as a specific issue.
Q365 Chair: Can I just pick you up again on this error? You say you only re-mark it if there is an error. That is a bit of a catch-22 or chicken and egg—whatever the expression is—because if somebody is predicted the top grade for a number of months doing mock exams and then gets very bad grades, surely if that person was re-marked you would only know if there had been an error if the paper was re-marked.
Sally Collier: In that instance, if you had a high-performing student who came out with a poor grade, you would of course then seek the script. You would say, “Something could quite easily have gone wrong here.” You would seek to get the script back and you would put in for review.
Q366 Chair: How would you make sure—again going back to my colleague’s point—that if the school or individual did not fund re-marks, that person would get some kind of justice?
Sally Collier: If that grade is changed, there is no charge.
Michelle Donelan: Yes, but that is a risk factor, is it not? If you are a low-income family and you cannot afford that risk, you are not going to do it. Equally, if the school is working to a tight budget, there might be a number of students in that predicament.
Q367 Chair: This is not the case of the 16 or 17 marks; this is the case of ensuring social justice for somebody who gets the grade.
Sally Collier: I completely understand that. We have a system where, if you have more resources, then you can allocate those resources to pay for different things. What we can do in that system is to make sure that the system of review is as fair as possible for all those accessing it.
Q368 Chair: Should there not be a red alert system for extreme cases like that, so when somebody who is predicted pretty high and then does pretty badly you would automatically re-mark?
Sally Collier: That is certainly something we can look at.
Q369 Chair: Would you be able to do that? Would you be able to have that kind of system?
Sally Collier: We would need to go away and look at implementability, looking at the pros and cons.
Q370 Lucy Powell: I have a very brief follow-up question on the impact on more disadvantaged children. Are you tracking the new GCSE and A-level system, which now relies wholly on exams that, as we have discussed at times, can look very different from what you are prepared for? In my experience, the children who cope less well with that are those who have less resilience, perhaps less self-confidence, who often are more disadvantaged children. Is that something you will track over the next two or three years, and will you go back to Government and say this is a trend, if you find it?
Sally Collier: Yes. We have an extensive programme of evaluation of all of the new reform qualifications. We are looking at which types of students responded well, which did not respond. Yes.
Q371 Lucy Powell: Are there any early findings of that from this last year that this is—
Sally Collier: I attended one of the teacher workshops myself. Clearly, before the reforms came into place, there was a lot of concern about how students would respond and how teachers would respond to teaching the new curriculum. In many of the subjects, teachers have enjoyed teaching the new curriculum, not having to repeat coursework constantly and re-mark and review. They have been freed up, and some of the teachers I have spoken to have been freed up from having to predict or being able to predict those grades in the way that they were used to. I had one English teacher who said to me, “We weren’t able to predict precisely the grade, so all we focused on was getting better in each.” She said her students had done even better than she had anticipated. Of course, we have had some subjects where there is much more content and teachers are struggling to meet that.
Lucy Powell: I am particularly talking about tracking those deprived children.
Chair: Can we just have slightly more concise answers?
Sally Collier: Sorry; mine are getting longer.
Q372 James Frith: Just on qualifications, are you concerned by an increasing number of unconditional offers being issued by mid-tier institutions, and what reason is there for this?
Roger Taylor: If I could answer, just personally, I think the NAO report was extremely interesting. It is an issue of concern, for the obvious reasons for that. From Ofqual’s point of view, it becomes particularly concerning at the point that it dramatically affects people’s motivation and the way that they approach qualifications, because that can start to undermine the validity of the qualification. If you have large numbers of people entering a qualification who, frankly, it does not really matter to them whether they get it or not, it can start to undermine the validity of that as a qualification, how the public understand its value. I do not think we are at that point, but I would agree it is something we need to keep a close eye on.
Q373 James Frith: What reason is there for this, do you think? That was the question, really. What reason does Ofqual see? I appreciate your personal insight.
Roger Taylor: I think it is a competition between higher education institutions for—
James Frith: Yes, absolutely. Thank you.
Emma Hardy: Can you agree, please, to consider having a deprivation-based charging system for re-marking? If you could look into that, I think that would be much fairer.
Chair: You said you are going to look at this red alert system particularly, but deprivation is an important point.
Emma Hardy: The deprivation link.
Q374 Ian Mearns: I was going to say something similar. We have used free school meals as a very broad-brush indication of deprivation within schools for many decades now. If there are pupils who are entitled to free school meals, that would certainly be a starting point for some sort of deprivation-based funding system for re-marking. Certainly that is something to consider. Has Ofqual undertaken any equality impact assessments on the effect of the new qualifications that you are introducing on boys and girls?
Sally Collier: Yes, we did—it was obviously before my time, but equality impact assessments were done. As my colleague Michelle, who is behind me, said, the perception about how well boys and girls do in particular examinations is often stronger than the reality. We did those assessments, and we are tracking, of course, the results as they start to come out.
Q375 Ian Mearns: Did the impact assessments go wider than gender? Did it do anything on ethnicity, for instance?
Sally Collier: Yes, and sexuality.
Q376 Ian Mearns: Were there any significant findings from your perspective on that?
Sally Collier: Four stakeholder groups put forward particular risks and concerns, and I think we very carefully tracked those when we were developing and accrediting the new qualifications. We continue to do so in all of our reform programmes. We are now talking to the equality groups and disability groups about functional skills, for example, so that is a constant feature of our work.
Q377 Ian Mearns: Would you be willing to share the outcome of those impact assessments with the Select Committee?
Sally Collier: Of course.
Ian Mearns: There has been a lot in here about sharing outcomes of impact assessments with the Select Committee. Thank you very much indeed.
Q378 Mr William Wragg: I am going back to the pre-appointment hearing again, Sally, I am sorry to say.
Sally Collier: Is music and maths coming?
Mr William Wragg: It is like Groundhog Day for you, isn’t it? We were discussing then the question of inter-subject comparability. After your appointment that was dropped by Ofqual. No? Have I got the wrong information there? You are still pursuing it?
Sally Collier: Yes.
Q379 Mr William Wragg: Could you explain how you are pursuing it and why?
Sally Collier: No, it is absolutely not dropped by Ofqual and it was not as a result of my appointment.
Mr William Wragg: I am sure it was not, no.
Sally Collier: It is a very important and intensive programme of work that we have carried out over the last year. We have said we are not going to have some grand aligning of all grades, where one goes up a bit and one goes down a bit, based on some notion of difficulty or severe grading. The board, under Roger’s chairmanship, has come to a decision that we will not have this grand aligning. What we have said is that where stakeholders have told us there are particular concerns about severe grading in particular subjects we will pick those off—
Q380 Mr William Wragg: Yes, but that is for particular subjects, isn’t it?
Sally Collier: Yes.
Q381 Mr William Wragg: Subject comparability has been dropped then, the grand realigning, if we call it that?
Sally Collier: Yes.
Roger Taylor: There never was a policy of a grand realignment.
Q382 Mr William Wragg: No, there was apparently a consideration of bringing that about.
Roger Taylor: We put it into our consultation as one of the options and 83% of people rejected it, I think for good reasons.
Q383 Mr William Wragg: For what reason?
Roger Taylor: We did say we could just apply a blanket rule of the same pass rate. Everyone felt that would not be the right way to go.
Q384 Mr William Wragg: Was it a question of an unfair value judgment or objectivity about the value to students, employers and educational institutions?
Roger Taylor: Yes, it would pay too little attention to the reality of the standard you need to achieve in a particular subject to be able to progress through whatever route that qualification would naturally lead to.
Sally Collier: We are looking at six A-level subjects—the three sciences and three languages. We are getting all of the subject-specific communities together to say to what extent this might be a severe grading issue versus all of the other factors that contribute to why students do or do not take up a particular subject.
Q385 Ian Mearns: You have recently published your interim report on teacher involvement in developing exam papers and other confidential assessments. You are now obviously evaluating the responses from that report. What is the right balance between exam integrity and quality of exam papers, and are you confident that you are striking the right balance now?
Sally Collier: Yes, we are. We have done quite a quick but intensive piece of work and called for evidence. We had some really good-quality evidence from across the piece. The response to the publication of our interim report is a pretty balanced set of judgments about what needs to be done. There are things we can tighten up, but the overwhelming response was that current teachers writing exam questions brings experience and quality, and if we took that away from the system that would be to the detriment of good-quality exam papers.
Q386 Ian Mearns: Are you at all worried that the evidence you gained relied heavily on evidence from awarding organisations and teacher examiners? Have you talked to anyone else other than those people?
Sally Collier: Yes, we have talked to a whole range of people, not just people involved in the writing of questions.
Q387 Ian Mearns: There is a concern that people with a vested interest would be saying things that would not be the case.
Sally Collier: Yes, of course. However, as the evidence from your previous hearing showed, when teachers were not involved in the writing of those questions and they perhaps were being written by higher education teachers, they were not targeted in the correct way for those students to answer. We have also looked internationally. We have looked at 12 other jurisdictions across the world and what they do. There are very similar features. Some do involve teachers and some do not. That is not to say we cannot strengthen the system to lessen the predictability of when a question comes up so a teacher writing an exam question would not know when their question is to appear.
Q388 Ian Mearns: What are the particular weaknesses of jurisdictions that do not involve teachers in writing exam questions?
Sally Collier: I would have to come back to you on that.
Q389 Ian Mearns: You say you have done a comparative analysis around the world, including jurisdictions that use a similar model and jurisdictions that do not. I would be interested to see what the weaknesses are of the ones that do not.
Roger Taylor: The primary weakness is the ability to produce questions that are unpredictable, that are manageable by a young person studying that subject and that test their skills and abilities in a way that they can engage with and display their skills effectively. People who are not engaged in teaching the subject to young people often find it quite difficult so you resort back to more predictable questions, or questions that are less effective at really testing the skills people want to test.
Q390 Emma Hardy: One of the other things I am concerned about is the disproportionate amount of people who write the examination papers coming from the independent sector. You can argue this is fair or this is not fair. To me, if you have a high number of people writing the paper from one area it does seem to give them an unfair advantage. Do you think there is something Ofqual could do to limit the number of people who have written that exam paper coming from particular independent schools or how many you could have from a certain school?
Sally Collier: We would certainly encourage teachers to become examiners. Perhaps it has changed over recent years from where it was seen as a course of professional development where you would become an examiner. We would certainly encourage that diversity. Of course, we keep track of where they are coming from.
Q391 Emma Hardy: Could you limit it and say there should not be more than a certain percentage of examiners from the independent sector?
Sally Collier: I am not trying to be defensive. What is going through my head as I answer that question is if we were to say you could not have a particular number from a set of independent schools, then we might not have any examiners for unique or specialist subjects.
Roger Taylor: This is another really interesting idea that we should take away, rather than pronouncing now whether we think it could work or not. We should absolutely take it away.
Q392 Michelle Donelan: Can I just follow up on that because I think it is a good point? When you say “encourage,” what does that actually mean in real terms? How would you go about encouraging? Could there be a reach-out programme?
Sally Collier: It is up to us and everybody else in the system, whether it is Ofsted or other bodies, to ensure that teachers are able to examine. We want more of that. The exam boards need many quality markers and examiners. It is up to us and also the wider system.
Can I make the point that it is also about protecting those teachers who are markers and examiners? We want them to do this. The vast majority of teachers in this space act with complete integrity and we want to put safeguards around them so they can do it.
Q393 Chair: Did you watch or have a chance to look at the evidence when we had the Cambridge Assessment Board and Eton?
Sally Collier: Yes.
Chair: Could I ask for your views about what they said, please?
Ian Mearns: Were you as shocked as we were?
Chair: If I can speak for the whole Committee, the feeling from the Committee was that Eton, on the whole, had taken action to deal with the problem but the Cambridge Board came across as unprepared and complacent. I wanted to take your views and what you are doing in terms of the issues that we raised as a Committee before that exam board.
Sally Collier: We obviously monitored very closely what happened in the case of that school. We are currently involved in an ongoing investigation about those practices.
Chair: We are talking in general terms regarding the Cambridge board. The Eton head teacher implied he was taking action to deal with it.
Sally Collier: That is rightly so. When identified, Cambridge did undertake its own investigation.
Q394 Chair: Are you satisfied with the Cambridge board?
Sally Collier: I would not be able to answer that with a specific yes or no. We are closely monitoring what they did over the summer.
Q395 Chair: Do you think there should be transparency in terms of the teachers setting the exams if they are at the same schools? Do you think there should also be anonymous exam papers—i.e. you produce some that are not used, there is a range of them and the teachers do not know what ones are going to be used?
Sally Collier: Certainly the latter, yes. I might come back to my point on predictability. We should be able to implement a system where a teacher does not know when their question is going to appear.
In terms of transparency, I know you had a debate about this at the last hearing. There are pros and cons of being completely transparent about everybody who is an examiner. We are trying to think of other areas of public life where someone has access to confidential information where you say, “Here you go; these people all have access to confidential information.” There are advantages and disadvantages and we need to weigh those very carefully. I know you talked about hacking, which is one element of the disadvantages.
Q396 Chair: I thought that was a complete red herring, to be honest. If Estonia can do a voting system online, I am sure we can work out an exam system for teachers.
Sally Collier: We can manage that. However, it is more that those individual teachers become a target, if you like. Does a parent say, “I want my child taught by the person who is examining”? You then get a focus on that teacher. There are advantages and disadvantages and we will look at that.
Roger Taylor: One thing to be clear about is that as part of this review we will look very closely at how that information is supported, how it is held and how we track it so we know precisely where teachers are coming from. The question as to whether it is made public or what format is made public is a different situation.
Q397 Chair: When will you have the results, for example the anonymised exam papers and so on that are part of your review?
Sally Collier: We expect to come to final conclusions in January.
Q398 Chair: Finally on this point, when issues or questions are raised about a particular exam board—as there have been about the Cambridge board and, as I said to you, the Committee is clearly unhappy with the evidence given—what are your mechanisms for assessing and investigating that exam board, and are you doing so?
Sally Collier: Yes, constantly with all of the main exam boards. We have a range of mechanisms. We have audits. We have formal investigations. We can require the exam boards to give us an undertaking to do something.
Q399 Chair: What are you doing specifically with Cambridge?
Sally Collier: I do not want to talk about that individual case where there is ongoing regulatory action.
Q400 Chair: Is that for judicial reasons? What is the reason you do not want to talk about it?
Sally Collier: You would not expect a regulator to come and talk about the specifics of an ongoing live case.
Chair: Thelma has been waiting incredibly patiently.
Thelma Walker: Yes, I have.
Chair: We are going to move on to national assessments.
Q401 Thelma Walker: I talked earlier about a seeming lack of public and stakeholder confidence in the changes to the assessment process. If we turn to early years and primary assessment specifically, do you think there is generally stakeholder confidence in the changes that have recently taken place?
Roger Taylor: You will have seen that we have reported on the national assessments process this year. We have confirmed that they are valid for the purposes of measuring the quality of primary schools. We have also identified a number of areas where steps could be taken to improve the validity of those tests.
Q402 Thelma Walker: Talking to teachers, in particular in the early years and key stage 1, one teacher said to me, “I feel like we are an ongoing experiment. I feel like we have just put something in place and then there is a new initiative.” For the changes to the testing system there were test materials that arrived late, teachers were ill-prepared and training had not taken place. At the end of the day we are talking about not only deprived children but all children. It is our children we are talking about, the next generation going forward, and teachers are discussing the idea that this is like an ongoing experiment that is almost at the whim of, “Let’s try this, so let’s roll this out.” This is what is coming through. There is a lack of confidence from professionals, now shared by many parents, that there is a lack of consultation and the timescales are too tight. What are your thoughts on this?
Sally Collier: The Government have consulted on primary assessment. We have a slightly different set of regulatory powers for regulating a Government agency than we do for regulating exam boards. We have been very clear about what our role is. We are interested in whether the qualifications are valid. We are interested in the accountability measures associated with those qualifications. We are consulting currently—if anyone still wants to respond to the consultation—on what our role is in national assessments. We have just produced the annual report of all of our work.
Clearly there has been a massive programme of reform in both primary and secondary. Those assessments do need time to bed in. There have been a number of tweaks made for the better that will continue to be made. I do not think that is necessarily a bad thing. For the first time you will learn where change is to be made, for example in the teacher assessment frameworks, and I think that is a good thing.
Q403 Thelma Walker: When you say the first time that we will learn, those kids have one chance in that year with that assessment with that idea of an experiment. One of the grammar tests was run in front of sixth-form English teachers who said, “Some of my students would find this difficult.” That is the kind of thing where the children are being almost experimented on with the level of the tests. It was acknowledged nationally that the reading test was too hard. What is Ofqual’s role in that?
Sally Collier: We do not set the content.
Thelma Walker: But the outcome.
Sally Collier: We do look at whether the tests were valid and whether they performed well. We have recently published our assessment of that reading test. There are some things we do before the exams are taken to make sure the processes for training of teachers and moderators is sufficient. When the exams have taken place we go back and do thorough evaluations of how those qualifications performed. We are completely transparent about where there are things that can be improved, as we have been with the reading test.
Q404 Thelma Walker: Can you tell me how you work with the Standards and Testing Agency? What is your relationship with it?
Sally Collier: We regulate it. As I said in a different way, we do not have the same range of powers that we do with other organisations. We obviously have a close working relationship with the Standards and Testing Agency.
Q405 Thelma Walker: Can you explain and unpick that for me? What kind of relationship? How do you relate?
Sally Collier: I have a dedicated staff working on national assessments and a team of experts. That team calls on our research colleagues. They will have regular meetings. They will ask for information about how processes are developing and what plans they have in place. We will scrutinise those plans. We will sit in and observe standards meetings, training meetings and awarding meetings. We will ask for information about how the qualifications have performed.
Q406 Thelma Walker: Can I interrupt there? If this liaison is so good, how did these inappropriate tests happen?
Sally Collier: I would caution against “inappropriate test” being used generically.
Thelma Walker: I am just quoting the stakeholders—the teachers.
Sally Collier: There were certainly particular concerns about aspects of the test. What we have found in the case of STA is that their three-year test development process, using a range of experts, is very, very robust, particularly with the examined elements in key stage 2. From our point of view, we are interested in validity. You will see from our annual report last week that we have been complimentary about how that test development process worked. There were things that were clearly picked up by stakeholders. With any new test there will not be perfection and some items will perform better than others. We aspire to perfection, as does the STA. When that happens we will point it out, make suggestions as to how it can be improved and we will then monitor what happens in the subsequent series.
Q407 Chair: I think both of you have been incredibly comprehensive. We have asked you about a huge range of issues, from the perception of A-level standards, exam integrity and confusion over the new grading system. I think my colleague Emma raised a very important issue of deprivation and fairness in terms of exam re-marks and the red alert issue. You have covered apprentices and technical education and now national assessments. We are grateful to you for your public service as well. We look forward to working with you over the coming months and wish you a very happy Christmas.
Sally Collier: And to you. Thank you.