Education Committee
Oral evidence: Accountability Hearings, HC 341
Tuesday 12 March 2019
Ordered by the House of Commons to be published on 12 March 2019.
Members present: Robert Halfon (Chair); Lucy Allan; Trudy Harrison; Ian Mearns; Lucy Powell; Thelma Walker; Mr William Wragg.
Questions 1840 - 1933
Witnesses
I: Witnesses: Sally Collier, Chief Regulator, Ofqual, and Roger Taylor, Chair, Ofqual.
Witnesses: Sally Collier and Roger Taylor.
Q1840 Chair: Good morning, thank you very much for coming in today. For the benefit of the tape and those watching our proceedings, could you kindly introduce yourselves and your titles please?
Sally Collier: Good morning, I am Sally Collier and I am the Chief Regulator, Ofqual.
Roger Taylor: Good morning, I am Roger Taylor and I am the Chair at Ofqual.
Q1841 Chair: Thank you. Could I please start off asking you what your current priorities are?
Sally Collier: Two main ones. First is developing our capability and capacity to regulate vocational qualifications. We have done a lot of work on that since we were here last, both in apprenticeship, external quality assurance, in T-levels and in functional skills. Next is all about ramping up that programme. Our second largest priority was ensuring, through the exam boards, the safe delivery of the reformed GCSEs and A-levels last year and, of course, we are in the third year of reform this year. They are our main two priorities, among a lot else, that I am sure we will cover during the course of the hearing.
Q1842 Chair: Thank you. A lot of the work our Committee does is on social justice, as you will know; addressing social injustice in education as well as skills and productivity. The Royal National Institute of Blind People’s 2016 report identified an average of 2.3 problems per Braille paper and 1.75 per modified language paper. I appreciate, of course, those are statistically low incidences, but it is not acceptable to be creating more difficulties for those who are at a disadvantage.
What can be done to ensure that fewer mistakes are made and also, particularly, past papers are rectified because obviously people are taught based on looking at past papers?
Sally Collier: I agree those statistics are worrying. In fact, we were already talking to the Royal National Institute of Blind People and other stakeholders about those numbers. They do not match our latest numbers for last year for errors; however that is not the point. We are going to have a detailed investigation into those numbers to see whether there is more that can be done with exam boards to reduce the occurrence of those errors.
Q1843 Chair: Could more rigorous conditions be introduced, for example in the tendering process, or is it the case you need to provide more resources, funding, to introduce additional checks?
Sally Collier: We do not know yet whether it is those things or it is a range of things within our papers that we could ensure exam boards undertake. Particularly with access to past papers, for example, if a student needs access to a past paper that is modified in some way, that should be available. Those sorts of things would be within our remit to do something about.
Q1844 Chair: When you say you do not know, why don’t you know if more rigorous conditions could be introduced? It is not a difficult question.
Sally Collier: If I could make a distinction; exam boards have to make live papers available in whatever modified format is required, whether that be large print, Braille or any other modification. What support exam boards provide for teaching and learning and for other types of materials is not within our rules. Nevertheless this is on the agenda for my next meeting with the exam boards, to talk about what we can ask them to do about support for blind children and children with other disabilities to make sure they have proper access to all of the materials they need.
Q1845 Chair: What are you doing to rectify the past papers issue? You look as if you are bursting to say something, Roger.
Roger Taylor: I think the past papers issue is an extremely interesting one. We have done a lot of work focused on errors and we have upped our regulatory activity with regard to errors. We are seeing, I think, a change in attitude at the exam boards and we are seeing more rigorous processes put in place to avoid errors.
The specific issue of correction of past papers is an interesting one, particularly the issue of whether modified past papers are provided in a corrected format. It is an interesting question and, as Sally says, is one of those areas we need to address and have a conversation about.
Q1846 Chair: The question is what are you specifically doing to rectify mistakes in the past and also to ensure it does not happen in the future? What are your specific actions, which I have not understood yet?
Sally Collier: Two specific actions. First, we will be getting underneath the data in that report as some things are errors and some things are best practice. First of all, we do want to get a handle on the data. Then we will be sharing that data with exam boards, raising the issue with exam boards and, if necessary, asking them to do more.
Q1847 Chair: That report was published in 2016. Therefore when you say you are looking at the data of that report, it is two and a bit years since.
Sally Collier: As I said, we have different categorisation of errors. We are only just working on the data in this report. We were working on it before it came out, but we were not aware of it two or three years ago.
Q1848 Chair: When will you be ready with some simple, clear guidance for the public or for the Committee about future and past papers to make sure there are not these kinds of mistakes?
Sally Collier: We are doing the work now, so shortly. I am not going to promise it is next week but it is not a year away, it is over the next weeks or months.
Chair: Given this was published in 2016 it is a long time, which is why I am asking.
Sally Collier: As the work is going on I am more than happy to be more specific for you and come back to you with a timeline.
Q1849 Chair: Could you set out for us what you are specifically doing? You have not made that clear. I am not trying to ask a difficult question. This is very important because we want everyone to have a chance to climb that ladder of opportunity. If this is a problem that has been going on for some time and there are problems with past papers, surely it should be your responsibility to act on it straightaway and not leave it.
Sally Collier: I certainly would not want to give the impression we have somehow been sitting on data, which we have not done. Once we aware of this data we want, as we always do, to understand it and then we can take action if we need to.
I am aware of the issue of access to past papers. I am aware of the issue of the difference in the numbers between what is a mistake and what is best practice. I do not have the specific numbers for past papers that then have not been corrected, which is why I cannot talk to you about a specific action.
Roger Taylor: To be clear, there is an issue around past papers and what exam boards are obliged to do. In terms of the future, it is very clear what they are obliged to do. The two things we have done are to up the monitoring of errors and to increase the obligation on exam boards to report them to us. We have started to take quite firm regulatory action and have increased our supervision of their mechanisms to prevent errors. Our ultimate sanction, if errors occur, is to fine exam boards. We have now started to use those sanctions in order to demonstrate that we are very clear this not acceptable.
Q1850 Chair: What are the sanctions?
Roger Taylor: Fines.
Sally Collier: There is a variety of sanctions.
Q1851 Chair: Who have you fined so far, which exam boards? What sanctions have you done so far?
Sally Collier: Last year we fined OCR £125,000.
Q1852 Chair: About this particular issue?
Sally Collier: Not about this specific issue.
Roger Taylor: No.
Q1853 Chair: Are you aware of any other impairment groups that are affected by access to exams apart from the Royal National Institute of Blind People and are you engaging with them?
Sally Collier: We are not aware of the same level of detail and analysis as has been done here. However, we are continuously engaging. We engage with The British Association of Teachers of the Deaf and the National Autistic Society. We have an access forum where, every year, those individuals can talk to us about concerns and we can then take action. If we need to, we can feed back. We have had particular sets of qualifications looked at when the reforms came in so these groups can tell us whether particular areas of the paper were a problem, and we then feed that back to the exam boards.
Q1854 Chair: Have any of these other groups expressed concerns?
Sally Collier: Obviously we need to continuously be on top of whether the students with disabilities have access to the papers they need. We have not had brought to our attention some of the specific issues in this particular report, which we were not aware of, but we are constantly on top of these issues.
Q1855 Chair: You are saying the only people who expressed concerns are the RNIB?
Sally Collier: The only numbers in research we have in relation to this issue is that report, but we are constantly in dialogue with all of these groups, which talk to us about the accessibility of papers.
Roger Taylor: To be clear, the engagement with The British Association of Teachers of the Deaf did result in recommendations around structuring papers that were disseminated to the exam boards. However, it did not lead to the same kind of specific issues you are talking about.
Q1856 Ian Mearns: Briefly on that issue, we have been lobbying quite intensively to try to get the GCSE in BSL. Are you aware of any progress on that?
Sally Collier: Yes, progress but the progress is with the Department developing the content. It has obviously been agreed that there will be a GCSE in British Sign Language. It is now DfE’s job to develop the content. When we can get our hands on the content, we can work on making sure it can be a proper accredited qualification.
Q1857 Thelma Walker: Can I ask if you are having any feedback or are sensing that the change from continuous assessment to summative assessment is disadvantaging SEND students or pupils?
Sally Collier: The reforms are still quite new. We did quality impact assessments when the qualifications were reformed. We started monitoring it last year. We will be monitoring it through this year and then revisiting those quality impact assessments to see exactly that—whether particular groups have been affected.
Q1858 Thelma Walker: That is ongoing at the moment?
Sally Collier: Absolutely ongoing.
Q1859 Chair: From your answers I do not get the impression that this is as if you are putting rocket boosters under it or that it is a major priority. You say you are looking at it. This has clearly been something that has gone on for some time. I was hoping you would say, “Yes, this is a serious problem. It is not on, and we are doing X, Y and Z to resolve it.” However, the answers have been, respectfully, very general. I could use other expressions for those answers.
Sally Collier: We hear your concern and we will make sure we set out fully exactly what we are doing in this space.
Chair: It seems unacceptable to me—
Sally Collier: It is, we agree with you.
Chair: —people do not have equality of opportunity because of a mess up—for which there is no excuse—from exam boards. It seems to me to be absolutely unacceptable. I do not feel any passion or get the view that you want to change it today rather than saying, “I am looking at the data” going back to a report from 2016.
Roger Taylor: To correct that perception, I do want to stress that the whole nature of a qualification is to create a fair basis on which to assess people against their skills. That is the entire purpose for its existing. We do focus on these issues and we do think very carefully about them. They are complex issues. However, I do fully accept your point that these unfairnesses are unacceptable and do need to be addressed. I want to stress that we do not take this lightly.
Chair: I hope you do write to the Committee with serious points of action.
Q1860 Lucy Powell: As you probably could have guessed, I am going to ask you about IGCSEs. Everybody behind you there is nodding. Thank you very much for your letter to the Committee outlining some of the issues.
As you have said in your letter, obviously the IGCSEs—which are still being sat predominantly by the majority of private schools—are largely unregulated by you, which I am not sure most people would know. Although I could see you were trying to be a bit equivocal, there is evidence they are marked in a different way. Indeed, because they are unregulated they are probably not subjected to the same comparable outcomes criteria that you use.
From the research done by the Education Datalab earlier this year it is pretty horrifying to any parent, myself included, to learn that 97% of those who sit the IGCSE in maths are getting a grade A* to C compared to just 71% of the new GCSE. When you look at the A*s and As, 68% of students sitting the IGCSE in maths are getting A* or A compared to 20% of those in state schools. Do you not think this is an absolute scandal?
Roger Taylor: Certainly it is a situation that is not really conducive to public trust in the examination system and it is a problem. On the Datalab research, I thought it was helpful that study was published. I should stress that we obviously are monitoring this and have researched it, because it is an issue of concern.
Our view would be that these qualifications are not systematically easier; however, they are not comparable and they are not regulated in the same way. That means there is a risk that a particular subject in a particular year will be easier and we do not have the mechanisms to do anything about that.
Q1861 Lucy Powell: The “easier” thing is subjective. I think it was the Government who described them as, “not as robust” and therefore potentially easier. What are you doing—because your job is to make sure there is that public trust—to talk to universities, to employers and to parents so these IGCSEs are not getting the same recognition? How can it possibly be fair that students who are applying to university—and at that point the only thing universities have to go on is their GCSE results—are being looked at in the same way when 68% of those sitting the maths IGCSE are getting an A* or A and for everyone in a state school sitting the new GCSEs it is capped at 20%? How can that possibly be fair?
Roger Taylor: First, it is not capped.
Lucy Powell: It is comparable outcomes, so therefore you are having the same number each year getting the same grades largely, aren’t you?
Roger Taylor: That is not how comparable outcomes work. Part of the mechanism is to use that information in setting the grading boundaries, to be clear, but it is also based on human judgment about the quality of the work done. There is not a cap.
Lucy Powell: No, but each year you are awarding the same numbers of grades across the piece, aren’t you?
Roger Taylor: No, under comparable outcomes that can vary.
Lucy Powell: Not significantly. You are not going to get 68% of people getting an A* or A, are you?
Roger Taylor: You would not expect it to vary significantly without clear evidence that the standard had changed substantially. However, we do have a mechanism—
Lucy Powell: There would be a total outcry if you suddenly awarded 68% of people A* or A.
Roger Taylor: Yes, it would be a very surprising result if that suddenly occurred. It is worth saying for direct comparisons of that sort there are differences in the cohorts taking the qualifications.
The key thing here is that under the system, if you are a private school in this country you can do whatever qualification you want and you can present it to a university. International Baccalaureate, for example, is taught in some schools. It does not create the same confusion because it does not use the same terminology and language. The most helpful thing here perhaps would be if the exam boards did not label their qualification—I understand why they call it an IGCSE because—
Q1862 Lucy Powell: What I am asking is what are you doing? Part of your job is about public trust, the public interface and the public understanding. What are you doing to say to universities in particular, and possibly employers as well, “This is not recognised and you should not recognise it because it is an unfair system”?
Sally Collier: Our job, of course, is to regulate those things that we are responsible for and to make clear what it is that we regulate and what we do not regulate. What we are doing in our communications—we are going to ramp up those communications again before this summer—is to be clear what is and what is not a regulated qualification. Clearly we have been talking to the Department about the system and what might be done about it.
Q1863 Lucy Powell: I do not know that is strong enough, to write to universities and say, “By the way, we regulate these and we do not regulate those”. There are deeper challenges there.
Sally Collier: Also that the standards are not precisely comparable.
Q1864 Lucy Powell: Yes, but also because they are not using comparable outcomes. The evidence is obvious, there is a massively disproportionate number of people—72% of those sitting the English language IGCSE—getting an A* or A. I have a 14 year-old in Year 10 at the moment going through these horrendous GCSEs. That would be a swim, wouldn’t it? It would be great if I thought he had a 72% chance of getting an A* or A in a different set of exams so he could apply to a Russell Group university without any consequence. It is a joke, isn’t it?
Roger Taylor: To be clear, the bulk of the difference you are talking about there is not to do with the exam but is to do with other social factors that affect people’s skill performance. I am not disagreeing with you about the basic point that there is a real risk of unfairness.
Lucy Powell: I am sorry, it is not just about that. You are saying, “They are a private school and therefore somehow they are more inherently clever”. I am sorry, I do not accept that.
Roger Taylor: I am not saying that at all. I am saying that the prior attainment of those students is significantly higher.
Lucy Powell: My son got a Level 6 in his SATs. He should be getting a Level 9 in his new GCSEs but they are really tough and difficult. He does not have a 72% chance of getting an A* or A, has he, under the new GCSEs even if he has strong prior attainment?
I would not try to defend these figures if I were you. I would be saying, “No, I recognise that is inherently wrong and I think there should be a stronger steer to universities not to accept these”.
Roger Taylor: I want to be really clear so we do not misinterpret these figures. You are suggesting that the scale of difference there reflects the degree to which they are easier, it does not. We need to be clear on that point. I accept the broader point you are making but I want to make sure we understand—
Q1865 Lucy Powell: I am not discussing whether they are easier or not. I am discussing that the chances of you getting an A* or A in English is 72% in the IGCSE and the chances of you getting an A* or A in the normal GCSE is 18%.
Roger Taylor: What I am suggesting is that the correct comparison would be between—
Lucy Powell: Those chances do not change. You are not going to suddenly award that to 72% of people in normal GCSEs.
Roger Taylor: The chances of somebody with a specific level of prior attainment achieving a grade on the new GCSE and an IGCSE—
Lucy Powell: You cannot compare that anymore, can you?
Roger Taylor: No, but in terms of the broad comparisons we have made, using the data that is available, it is clear we cannot guarantee that—
Q1866 Lucy Powell: They are so skewed now you cannot do that. Can I ask you—sorry, I know other colleagues want to come in—what you are doing to say to universities, “You should not be recognising these because they are not regulated, they do not meet all our other criteria and they are not being judged on comparable outcomes”, which is a critical issue here?
Sally Collier: I will try again. What we are doing is being clear about what we are responsible for, we are not responsible for regulating another type of qualification and we cannot make precise comparisons between the two. What is taught in independent schools and what is accepted by the system is not a matter for us to decide.
Q1867 Chair: Is it fair to have one set of rules for private schools and one set of rules for state schools, because state schools are not allowed to do the IGCSE? The fact is whether these are apples or pears or not is immaterial to employers—employers just see the word “GCSE”—there and clearly there is not a level playing field.
Roger Taylor: That is exactly the issue. I do not think the correct answer is to say, “Let’s allow IGCSEs to be taught in any school” because the whole point of the reform programme was to prevent there being a race to the bottom of standards and to apply much greater rigour to GCSEs to ensure that they genuinely were comparable. That has been achieved. For exactly the reasons you suggest, if IGCSEs are treated the same as GCSEs and then we allow them to be taught through the system we will be back exactly where we started, with a potential race to the bottom.
Lucy Powell: They are being treated the same by universities and employers so you do have a two-tier system that is deeply unfair.
Ian Mearns: It is the currency that they hold in the system of application. They hold a currency by giving it equal value.
Q1868 Chair: Is it fair to have one set of rules for private schools and one set of rules for state schools in terms of all this, yes or no?
Roger Taylor: No.
Sally Collier: No.
Q1869 Chair: Have you made that clear to the Department?
Sally Collier: Yes.
Q1870 Chair: What have they said?
Roger Taylor: They are exploring. We also obviously have legal routes around the use of the term “GCSE” and its use in names. We are exploring what the mechanisms are that could be used to address this and the Department is also looking at it.
Q1871 Mr William Wragg: As the regulator you have a responsibility not only to those learners but also higher education institutions and employers. Surely that would cover comparing the two.
I take entirely the point you make about the cohorts, prior attainment and so on. One way of surely taking the sting out of this—I do not know if you have done this yet—is a straightforward direct comparison of the two syllabuses, the marking criteria and the grading criteria for each new brand GCSE and IGCSE. Has that been undertaken?
Roger Taylor: That has been undertaken.
Sally Collier: Yes, for the legacy GCSEs we did it for English and maths. We published our research for that. It is very difficult to do that—as Lucy pointed out—for the new reformed GCSEs because we do not have the prior attainment GCSE data. These things are different in terms of their assessment structures.
Q1872 Mr William Wragg: Surely in terms of, for example, mathematics the content is very straightforward to compare, not just for the legacy GCSEs.
Sally Collier: It is obviously easier in mathematics. However, the benchmark, if you like, for making precise comparisons is very high in the regulatory world. We have said we cannot make precise comparisons because of the nature of the different constructs, the different content. We have said we are not able to do that.
Roger Taylor: On a broad comparison you would not be able to say clearly that one is easier or harder than the other. The sense of unfairness comes from the fact that it is not held to the same standard in the setting of grade boundaries. That is where the unfairness lies and there is no remedy to that.
Q1873 Ian Mearns: We have set out the ground here from the DfE, Lucy has examined it and the DfE thinks they are less robust. There is a feeling they are inherently unfair because it is a less robust qualification. Is not the fact we have this separate examination system for the independent sector reinforcing a whole range of other privileges that exist there?
Roger Taylor: That is precisely why this is such a disturbing issue in terms of the fairness of the system, because it is used in one sector of the education system and not used in another. You have absolutely captured the issue here.
Ian Mearns: We have to get the message out there much louder and much clearer that these should not be treated as equivalent with GCSEs.
Q1874 Chair: Basically what we are saying is that if you are wealthy enough to afford to go to private school, on top of that you are going to get an easier exam that is called the same name and recognised by employers. However, if you are not wealthy you go to state school and do a higher quality, higher standard exam, also called GCSE, and potentially get lower grades even though that person from the private school is having all those advantages.
Roger Taylor: I agree with that. However, we also have to bear in mind that as a regulator we need to speak carefully and recognise the political decisions that have been made by Parliament that do allow for the provision of unregulated qualifications, do allow for private schools to teach whatever qualifications they want and do allow for universities to assess those qualifications. We need to be careful. We recognise the issue you are talking about but we do need to recognise that those issues do result from a set of arrangements that have been approved by Parliament.
Chair: I am not saying get rid of them. I am saying that state schools should have the same rights; that is all.
Q1875 Lucy Powell: Finally on this, the other thing that frustrates me is that it took me doing lots of parliamentary questions about this, because I could see what was happening in schools, to bring this to the public’s attention. Surely it is your job, alongside the DfE, to bring more attention to this issue. The fact, as I say, you have a 72% chance in English language or a 68% chance in maths of getting an A* or A is a complete outrage. Why has it taken parliamentary questions from a MP and this Committee now talking to you to bring these things to the public domain? Why did people not know about this?
Sally Collier: It is our job to regulate effectively—
Lucy Powell: It is your job also to make sure there is trust in the system. There cannot be trust in the system when there is gaming going on.
Sally Collier: We have published research on this matter. We have communicated about what we do regulate and what we do not regulate. We will continue to highlight the differences between the two qualifications.
Lucy Powell: I have not seen any of that as a Member of Parliament, let alone as a parent.
Q1876 Thelma Walker: Could I ask you to focus on public confidence in our examination system and the integrity of our whole system? I know that malpractice is relatively rare but what are you doing to address that which exists?
Sally Collier: Malpractice, a lot. As you will be aware, when we came last time we were just about to launch our consultation on teachers who were question-paper writers and examiners with access to confidential information. We had that large consultation.
Let me start by saying there are probably three or four types of malpractice. There is malpractice by a student who may take a mobile phone into an exam room. There is malpractice by teachers who may give too much support, for example, to students on coursework or be question writers who may share confidential information. Then there is malpractice by people associated with exam offices potentially having access to confidential information. We have had strategies on all of those three groups of people.
For students we have had a campaign, “Please don’t take your mobile phone into the exam room”. Many of them—there were over 1,000 penalties last year—were done inadvertently; taking a phone into the exam room is not always deliberate. We have had a strategy of communication for students.
We have had our consultation on teacher examiners. The new rules that we are putting in place from 2020 should ensure that no question-paper writer knows with any certainty whether their question is going to appear in the next series of exams. We have put safeguards in place for schools in terms of conflict of interest registers. We have had a strategy for teachers and now we have a strategy for exams officers, making sure they are absolutely clear what it is they can and cannot do.
Q1877 Thelma Walker: Is it right that you chose to allow awarding organisations to decide whether or not to publish information about the people who are involved in the setting of our exams?
Sally Collier: We considered this very carefully at the time whether these names should become transparent. It is a balance decision. There are pros and cons of doing that, as we discussed last time.
Q1878 Thelma Walker: Is it not passing the buck though to say, “You can either publish it or not”?
Sally Collier: No, not at all because we think the really strong safeguards that we have now put in place, to make sure that you cannot predict with any certainty when your question is going to appear, will do the trick.
Q1879 Thelma Walker: The fact that awarding organisations can choose whether to publish who has been involved in the setting of the exams, surely that is not transparent to the public?
Roger Taylor: From the point of view of the exam boards, there are risks both ways. If you publish the name of somebody who is involved in a confidential process to produce examination materials, there are risks associated with that as well as benefits. That was the issue that needed to be considered.
Q1880 Trudy Harrison: Last time you were here you talked about being willing and ready to do more in terms of responsibility for external quality assurance on end-point assessments with apprenticeships. Can you talk about our report and that aspect?
Sally Collier: We have spent the time since we were here developing our work in this area. We have new rules. We are the external quality assurer of the 61 standards. I think we have done a good job in proving that as the regulator we can do this job and we can do it well. I have feedback from employers and from awarding organisations that says we can do a good job. Therefore we are still in the same position as we were a year ago, but we are even more ready to take on a larger role.
Q1881 Trudy Harrison: I think it is fair to say that the system for overseeing apprenticeships is already quite complex with DfE, ESFA, training providers, Ofsted for Levels 2 to 5 and then the Office for Students for Levels 6 and 7, Institute for Apprenticeships on technical information, end-point assessment organisations, then the employers and, of course, the apprentices. Is it not going to add more complexity to an already complicated system? What value could you really bring? What would be our measure of success?
Sally Collier: Is our role adding success? You are absolutely right that it is complex and confusing in places. What role could we bring? I think we can bring some clarity to the process. Having end-point assessments independently regulated, in the same way that other types of qualification are, I think would help the public trust and help confidence in the system.
Q1882 Trudy Harrison: As a parent—as I am of four daughters aged between 16 and 20—we are looking at this at home. How will I understand how the system has improved? That public confidence that you talk about, what will it look like?
Roger Taylor: The first thing would be the experience of apprentices going through these qualifications and their outcomes in terms of getting employment. The role of regulation in that is ensuring that standards are maintained, the end-point assessment is a high-quality assessment that establishes that the young person who has gone through this training has acquired the skills that were intended and is now able to pursue the career they wanted to pursue.
Regarding the mechanisms we have to do that at the moment, I think there are understandable concerns that we do not have consistency of our mechanisms to ensure we are achieving those standards.
Q1883 Trudy Harrison: How will it be measured? At the moment it seems difficult to understand how that satisfaction rate is measured, certainly from an employer’s perspective.
Roger Taylor: The first element of it is a technical evaluation of the end-point assessments themselves and the way in which they are administered, just in the way that other qualifications are regulated, which is just to ensure the appropriate processes are in place, the appropriate thinking has been done about how the assessment is going to be conducted, that the assessment materials themselves are fit for purpose and that the process is managed in a professional and reliable fashion. That would be the first objective.
Beyond that, in terms of other measures, there are a number of possible routes. The first thing we want to establish is clear standards for what a good end-point assessment looks like.
Q1884 Trudy Harrison: Finally, you said you were already up to 68 standards?
Sally Collier: Sixty-one.
Trudy Harrison: Sixty-one, of how many?
Sally Collier: Over 400 now, I think.
Q1885 Trudy Harrison: To take it further, what would need to be done?
Sally Collier: We are now confident in our process of looking at assessment plans, looking at all the materials for the end-point assessment and working with employer groups. If we were to expand, we would need more people and clearly more resources. We think we have a robust system that those that we regulate seem to appreciate. Clearly we would need more resources in order to be able to do that.
Q1886 Chair: Are you confident the IFA has the institutional knowledge to assess quality when they have only been in the field a couple of years and have no track record?
Roger Taylor: I think the IFA understands the issues and is addressing them. I would prefer not to comment on other organisations, I think that is probably best addressed to them.
Q1887 Chair: The reason for my question is—as Trudy has touched on in some ways—why do other organisations and trade bodies need to be involved in the quality assurance of apprenticeship exams when no other qualification works this way? Why isn’t it just you?
Roger Taylor: The IFA does have a clearly differentiated role in terms of engaging with employers about—
Chair: I know the role. I am asking about why you need to have the IFA and a whole load of trade organisations. You have more organisations doing the end-point assessment; it is like a “Ben Hur” movie, there is a cast of thousands involved in it. Surely it should just be you with the expertise that you have. You could have providers feeding into Ofqual separately, given their knowledge, without being directly involved in the quality assurance of their own home, for example.
Sally Collier: We would agree that we think we have a long track record of expertise in assessment of quality qualifications.
Q1888 Chair: You have the resources to do it?
Sally Collier: We have the resources to do what we currently do. We would need more resources if we were to expand. Clearly the Institute has a much broader role than the quality of the end-point assessment in terms of being the voice of the employer.
Q1889 Chair: You think it is unnecessary for all these other bodies to be involved and you could do the job pretty competently, yes or no?
Roger Taylor: Yes.
Sally Collier: Yes.
Chair: Thank you.
Q1890 Lucy Allan: I want to ask about the reforms to BTECs and to understand from you whether universities and employers know the difference between the new and old version of the Level 3 BTEC. Roger, could I start with you on that?
Roger Taylor: This is an issue and it is not dissimilar to the IGCSE issue. The first thing I would say is that I think the reforms to BTECs are improving standards. There was an expectation that when the old-style BTECs were taken away from performance tables they would cease to be used. That has not come about. I agree this is a situation that creates confusion and we fully support the moves by the Department to cease funding for the old-style qualifications. Where we just have one version of that qualification is where we need to get to.
Q1891 Lucy Allan: Universities and employers are confused between new and old. Is that what you are saying?
Roger Taylor: They do know the difference. They are not using the information as much as we might expect them to in determining allocation of university places.
Sally Collier: We had a concerted effort last summer to talk with many higher education institutions about the differences between the two. This is, of course, a situation where we regulate both sets of the qualification. We did have a campaign and will have another campaign for those that are left this year. However, as Roger said, we support the Department’s move to rationalise this landscape.
Q1892 Lucy Allan: Roger, you said that the reforms to vocational qualifications gave Ofqual an opportunity to make sure that quality is designed in from the outset. Do you think the reforms achieved that?
Roger Taylor: Yes. I should distinguish between a full-scale reform of qualification in the form of a T-level, which allows us to set a new standard, involves much greater independent scrutiny of how the assessments are made and defines a new set of standards. In particular, we can apply special conditions to that type of qualification. That is rather different from what has happened with the applied generals, where essentially performance table rules have come in. That would not qualify as a full-scale reform that allows the wholesale building in of quality from the outset that T-levels do allow for.
Q1893 Lucy Allan: You are satisfied with what the reforms have achieved with BTECs?
Roger Taylor: The new applied generals certainly are more rigorous and more manageable in terms of ensuring standards than the old style; that is true.
Q1894 Chair: To follow on from what Lucy is saying on the BTECs; I was just looking at figures from The Higher Education Statistics Agency and see that if you take a student scoring less than C-C-D in their A-levels they are far less likely to drop out of university than a student who has achieved the highest possible BTEC grades. As you know, there is an incredibly high drop-out rate for BTEC students. What is being done to ensure, as far as you are concerned, the BTEC is a rigorous qualification that prepares students for university and stops or curtails the drop-out rate, if this the route they want to follow?
Sally Collier: We have done a number of things. We published some research before Christmas, again shining a light on the issue of standards in the legacy applied general qualifications. It is absolutely unacceptable that a student going into higher education with different types of qualification should have a lesser outcome or their qualification should be less valuable than those doing A-levels. We have done our usual, really intensive, piece of research. We have clearly advised the Department that the best way to build in standards and the best way to build in confidence is to have reform from the start, as with T-levels, as with the GCSEs and A-levels. The new reformed BTECs have had partial reform. They are more rigorous in some respects and we will be doing the same tracking with the new applied generals as we have done with the old ones.
Q1895 Chair: You are tracking the drop-out rates, are you, from higher education?
Sally Collier: We will be looking at the outcomes for reformed BTECs against a control group of those with A-levels.
Q1896 Chair: When will that be published?
Sally Collier: It is a future piece of work.
Q1897 Ian Mearns: T-levels have been mentioned. Are you satisfied with your involvement in the new T-levels so far?
Sally Collier: We are very pleased to have been appointed as the independent regulator for the technical qualification within the programme. I do not think we had been when we were here a year ago. We have worked intensively with the Institute over the last 12 months to align their contract management processes with our regulatory processes. Every deadline for the first tranche of T-levels has been met. We now have two firms that have been awarded the contracts so for the first tranche we are in a good place.
Q1898 Ian Mearns: What will your role be within that dual competence issue with the Institute of Apprenticeships?
Sally Collier: We will accredit or not the qualifications, just as we have with GCSEs and O-levels.
Q1899 Ian Mearns: I am afraid to say I am bit of a novice on T-levels. I understand they comprise about five different experiences or competences, all of which must be completed and passed so the whole qualification can be gained. That sounds quite a complex process. What would you say the equivalence would be in old money of pre-existing qualifications?
Roger Taylor: I do not think there is an equivalent. The T-levels programme is an innovative programme that tries to define quite a broad programme of study, which includes things such as work placements and various aspects of the experiences that young people go through. It has a different approach to the qualification elements within it. It does involve a collection of qualifications; you have to demonstrate English and maths at a certain level as well as the elements of the technical qualification within the T-level. There is no equivalent to that in either old money or in general qualifications.
Q1900 Ian Mearns: When you say there is no equivalent, you cannot even look at a combination of other factors and say it would be appropriate to weight it the same as that?
Roger Taylor: No, I really think it is quite a different animal.
Q1901 Ian Mearns: In terms of the currency that a T-level will have in the world of work or going onto further qualifications, how difficult is it going to be to get employers or higher education institutions to understand what competencies and qualifications the individual who passes has?
Sally Collier: It is a very big communications job by all of those, not least the Department. We play a part and I hope that our role in regulating the qualification within the T-level is one that will bring confidence to stakeholders in the programme. You are absolutely right that it is a much bigger job, so parents and employers feel they are able to select this qualification. It is a fantastic opportunity to reform these qualifications.
Q1902 Ian Mearns: Do you think T-levels, the few that have been prepared already, are ready to go and that students will be able to engage with them fruitfully within the first year?
Sally Collier: They are not ready to go. The contracts have just been awarded. We now await materials to come through from the awarding organisations. Remember, our role is within the qualification for the overall programme.
Q1903 Ian Mearns: Surely before an individual young person embarks on trying to gain a qualification such as that, you, as the regulator, should be clear and have a fair understanding that the courses the young people are going to be engaged in are ready to go and are fully fledged?
Sally Collier: Of course. The risks associated with readiness in schools and colleges is with the overall T-level programme in the Department. What we have to do is make sure that good-quality qualifications are available in good time. What the winning contractors have to do is assure us that their materials are of sufficient quality so that we can launch the qualifications.
Q1904 Ian Mearns: Are you clear that you will not sign off a course for a young person to embark on until you are absolutely confident that they will get a good-quality experience?
Sally Collier: What we are clear about is that we will not sign off a qualification, the materials and the assessment materials, unless it meets our quality bar.
Q1905 Ian Mearns: Have you been working with the IFA to implement the system?
Sally Collier: Extremely closely.
Q1906 Ian Mearns: Could you elaborate on that a little, please?
Sally Collier: Working out—you mentioned the Institute’s role—what particular roles in the assurance of quality are for us and what roles are for the Institute. We would not expect the Institute to be going into great detail about material supporting a qualification, what the questions are, whether they cover the full range of grades, whether they cover the entire content. We have been delineating roles and responsibilities so we can get on and do our job.
Q1907 Chair: The IFA has put the QA out to tender. Are you bidding for that?
Sally Collier: No.
Q1908 Chair: Why is that?
Sally Collier: It would not seem appropriate for a regulator to bid for a contract with another part of Government.
Q1909 Mr William Wragg: What lessons have you learnt from the implementation of the reformed single and combined science GCSEs?
Sally Collier: Let’s start with the combined science. One of the lessons we learnt from the first award of the new combined science last year was that there looked to have been some inappropriate ‘tiering’ entry for students on combined science. Had we not stepped in last summer many more students would have fallen off and become ungraded in the higher tier. We have had much more communication with schools. I have written to every head teacher about ‘tiering’ in combined science for this summer.
Roger Taylor: The other aspect it is perhaps worth drawing attention to is that we have done a lot of work on the approach to evaluation of people’s practical skills and our recommendation to do that primarily through examination but to require young people to do more actual experiments. It was a contentious decision; we did a lot of research into it. We are finding that it is working in terms of assuring that young people do have those practical skills and, indeed, have a degree of more freedom to experiment and understand science properly.
Q1910 Mr William Wragg: Clearly for the double award, you can get two equal or adjacent grades. With a scale of 1 to 9 there is a possibility of 17 different grading outcomes. I believe a number of science bodies and science teachers raised concern about what constituted a ‘strong pass’, was it a 5-4 or a 5-5. It sounds like we are reading the football results. Do we have an answer to what constitutes a ‘strong pass’, is it the 5-4 or the 5-5?
Roger Taylor: In Ofqual’s world, any grade you get is a pass—a 1-1 is a pass—you have your GCSE.
Q1911 Mr William Wragg: What is the average then? What is the benchmark grade, which the grade C was under the old legacy GCSEs?
Roger Taylor: A C maps to a grade 4, and in combined science, it will be a 4-4.
Q1912 Mr William Wragg: I think you mentioned last time, Sally, that Ofqual was going to undertake an awareness campaign as to the new GCSEs among parents and particular groups of employers. What groups did you target it at and what form did that take?
Sally Collier: Up until the new GCSEs were introduced, we ran and paid for a specific targeted campaign. In our perception survey last year that targets particular groups, which happens every year, I am very pleased to say that if you look over the three-year period there has been a rise in all of those groups, including employers. Clearly these are big changes and they take time to work through the system but we are very pleased with the improvements.
Q1913 Mr William Wragg: Could you tell us what those improvements are?
Sally Collier: The latest number for employers is 68%, it was 38% when we started our awareness campaign. The latest figures I have are for April last year so we have another 14 months. We are just about to publish the perceptions survey again, so you might expect they have increased again. We have had much more engagement over the last 12 months with groups of employers, whether that be on platforms or our own particular conferences. Those awareness figures are rising but this is a big job and it takes everybody in the system to communicate it.
Q1914 Mr William Wragg: Of course and it will take time, I accept that. What about parents?
Sally Collier: For parents I think the last number was 80%, again a big improvement from three years before.
Q1915 Mr William Wragg: As we have mentioned the perception survey, the old-record question I tend to ask quite often—I can see you wincing as it approaches—is when you as a regulator have said, “No, Minister” and no to Government. Has there been anything in the last year that has been notable for you to say, “No” to Government or, “Go back and think again, Government”?
Sally Collier: Yes. What I would say is I think we demonstrate our independence in a number of ways.
Q1916 Mr William Wragg: Can I ask, for that “yes” what is the example?
Sally Collier: There are a number of occasions, particularly on T-levels and timelines, and particularly on the new reformed applied generals and the safety net that we felt was very important for students taking those new qualifications last year, areas where perhaps the Government has changed its approach or has listened to our advice.
Q1917 Mr William Wragg: I think it was 2015 that the perception survey removed the question about whether respondents thought Ofqual was too close to Government. I think last time you mentioned monitoring what public perception was to that particular question through other means rather than asking it in a survey. How have you been monitoring that?
Sally Collier: I ask the question of everybody Roger and I speak to—I meet a lot with teacher unions and meet with all sorts of stakeholders—whether they have any concerns with our awarding bodies. As I think I said last time, we would get a sense if there was concern about where we were not speaking up or where we were not giving that strong advice. As I said earlier, I think we demonstrate our independence. We have published over 20 research reports and 19 consultations, we are very transparent about what we are doing and the decisions we are taking.
Q1918 Mr William Wragg: Do you think there is a value to perhaps adding that question once more to the perception survey?
Sally Collier: That survey is all about the outcomes and the qualifications that students get and not about us. I would not necessarily expect the man on the street to know the intricacies of the regulator. We are not the same as Ofsted.
Q1919 Mr William Wragg: It would not be the man in the street responding to that questionnaire, would it?
Sally Collier: There is the general public in there.
Q1920 Mr William Wragg: With due respect, obviously you accept this, it is not the issue of the moment necessarily, is it? Those stakeholders—other employers, educational centres and so on—they would understand, to some degree at least.
Sally Collier: Of course, the stakeholders in the system. I am happy to look at it again.
Mr William Wragg: Thank you.
Q1921 Lucy Powell: Following on from that, in terms of pushback on Government potentially around exam stress. You recently published a guide for students on coping with that stress. Is that part of your remit, do you think, the impact on students?
Roger Taylor: I think any organisation that is involved in education or with young people has to take extremely seriously the evidence of growing levels of anxiety.
Q1922 Lucy Powell: How are you doing that?
Roger Taylor: The first thing is we are taking very seriously the suggestion that the structure of exams themselves is contributing. However, the evidence does not support the view that it is the way exams are designed that is the issue.
Q1923 Lucy Powell: What evidence is that? We recently did a joint inquiry with the Health Committee on young people’s mental health, obviously rising self-harm and rising teenage-suicide rates. Surprisingly to us the top issue that was raised by the young people we spoke to was exam pressure, it was not social media or other things that people might perceive, so what is your evidence for saying that?
Roger Taylor: Exam stress has consistently been very high, and that has been true both before and after changes were made to these qualifications. If you look internationally, there is no connection.
Q1924 Lucy Powell: Except that young people we spoke to, and many of the teachers we spoke to, as part of that inquiry and elsewhere, the sheer number now of exams, because it is all exam-based, in a short period of time, would totally stress out the most resilient adult. The exam timetable is much heavier now, isn’t it?
Roger Taylor: The amount of time spent under exam conditions has not increased because controlled assessment has been taken out of many qualifications. In many cases, it would not be true to say people are spending more time under exam conditions than they were with the old qualifications.
Q1925 Lucy Powell: You are using your wording there quite carefully. There are many more exams at the end, where everything is at stake, in that short space of time, as opposed to exam conditions over a longer period. Can you keep an eye on that and if you see that--
Roger Taylor: Yes, absolutely.
Sally Collier: You had Andreas Schleicher here, didn’t you, and he says that we do not have much evidence to support that tests drive student anxiety. There is no correlation across countries between the prevalence of tests and student anxiety. Of course, with the change, students are talking more about stress and anxiety, and with the consequences that we hear about and that is why we have stepped into this space, but what Andreas Schleicher says the issue is about is whether students feel supported in this change. When we look at causes, the actual test is a feature, but the way that students feel pressurized, and what else is going on in the system, are important contributory factors.
Lucy Powell: Sure, I would agree with that.
Q1926 Thelma Walker: Picking up on what Andreas Schleicher said to our Committee, he also said if you focus just on what you can measure, you sacrifice validity gains for efficiency gains and you sacrifice relevance for reliability. He also said that. Ofqual seems to be valuing that exams are more efficient and reliable but surely they are less valid and relevant to students’ lives. If young people cannot see the relevance of what they are doing, it impacts on their attitude to what they are doing. It begs the questions why are we so insistent at the moment about using this kind of intolerant, unsupportive exam system.
Roger Taylor: A young person can be supported through different sorts of assessments. I would not agree that one particular form of assessment is less able to be supported than another. Validity is at the heart of what we consider. If you are saying that, for example, controlled assessment can be more valid—computer science is an interesting one where we have just done an evaluation on exactly this issue. People were very keen to support young people to do joint project working, and it is important to do that as a part of education, but if ultimately you cannot assess validly in some form of qualification, that aspect of it, it is better to simply not do that. Our job is to assess what can be validly assessed through an assessment process. I completely agree with you that that is not the sum of what an education is but we should not undermined the validity of assessment because we wish to encourage young people to learn in different ways.
Q1927 Lucy Powell: We talked about the student experience when you were last here, which relates to this, given the very low mark that was needed to pass the maths exam because it was so tough, mistakes that were made in the English exam, obscure characters being the main point of English tests, and so on, rather than the main thrust of a text. You said you would have a look at that point, the student experience, that comparable outcomes iron it out, but what does it do in terms of motivating those students to continue studying those subjects? What have you done on student experience?
Sally Collier: We have spoken to over 500 teachers across the subjects about the way reformed qualifications have been taught and the way that the examinations played out.
Q1928 Lucy Powell: Students?
Sally Collier: Yes, when our people go out into schools and colleges, but clearly teachers also tell us what their students are feeling and experiencing. Where we can, and where there are particular aspects that we can feed back to exam boards, in order to make some changes. I totally agree with you about the student experience where they can only access a tiny proportion of the paper. We need to get tiering decisions right. Where a student may have been much better being able to access more of the exam on the foundation tier, we have been clear, and I have written to every school about tiering decisions.
Q1929 Chair: Any report on mental health studies is very good and important to do, but why is it your role to do that? It is quite outside the role of regulator of qualifications. Did you ask the DfE to do it? Surely it should be the responsibility of the DfE, or possibly Ofsted.
Sally Collier: Yes, of course, all of those people. Because this is such a big issue and an important one, we felt we could help. We felt we could use our expertise, particular with the academics at Manchester.
Q1930 Chair: Did you do it independently or did you ask the DfE first?
Sally Collier: We did it independently.
Q1931 Chair: You did it independently. I am not against that, I am just trying to understand.
Given that students now, for the most part, stay in education until the age of 18, unless they are doing an early apprenticeship, as an exam regulator, do you see as much importance in having significant exams at the age of 16?
Roger Taylor: That is a very good question. It is clearly a policy issue. It is not something that Ofqual would comment on, as to whether this is the right answer or wrong answer.
The only thing we might point to is the scale of that kind of change to the education system. That is not a reason for not making the change, but our main observation would be that it would be a big change.
Q1932 Chair: What would you think of the possibility of having a wider curriculum at the age of 18, and not narrowing down A levels? The question I am trying to ask is whether or not our current exam and qualification system is relevant in providing the necessary skills, given the march of the robots, the fourth industrial revolution, which is something we are doing a separate inquiry on.
I am asking you to not give me a politician’s answer. Just tell me what you think, what you really think. Tell the Committee what you really think.
Roger Taylor: I don’t think I could speak for Ofqual on that question. Personally, however, I do think that there are aspects of education in terms of a young person’s understanding of what is going to happen when they leave education, what life is going to be like, that would be of increasing benefit to young people as employment becomes less certain. It is an absolutely appropriate topic for policymakers to focus on.
Q1933 Chair: Can I get you to say you think there should be at least a national debate and discussion on the future of our exam system, whether or not we need GCSEs and whether or not we should widen our A levels to have a wider baccalaureate at 18?
Roger Taylor: We are having a national debate and that is entirely appropriate, yes.
Sally Collier: I completely agree.
Q1934 Chair: Going back to the very beginning, to my questions on the exam qualifications of the Royal National Institute of Blind People, I do feel that everyone must be on a level playing field and to end, I want to read you a quote from a witness. We had a witness to our Special Educational Needs inquiry, who said, “What gets me is that even if a blind child gets the specialist support, when they get to national exams, can you believe that on your average science GCSE, there are six Braille errors that have a major impact on your ability to answer the question. That is on average, in every single science GCSE. On average in GCSE exam papers there are two major Braille errors, high-impact errors. Even if you get to your GCSE, you get a GCSE paper where you cannot do two or six of the questions. My son is 11, so I have five years to try to fix that problem.” This is a serious quote from evidence to our Committee in our inquiry on children with special educational needs. This is wrong. It is very wrong, even if it affects a small proportion of pupils. I am stressing this to you, because I didn’t get from your earlier answers, that this is something you should treat as a major priority, if you do believe as a regulator in equal opportunity and giving everyone the chance to climb the ladder.
Sally Collier: We do hear you and we will come back to you. We agree with you, we hear you, and we will come back to you on exactly what we are doing and when we are doing it.
Chair: Thank you very much for sustaining some tough questioning and for your public service. It is very much appreciated by all of us. Thank you.