Business, Innovation and Skills Committee
Oral evidence: Assessing Quality in Higher Education
HC 572-ii
Tuesday 1 December 2015
Ordered by the House of Commons to be published on 1 December 2015.
Witnesses including written evidence where submitted:
At 10.00am:
At 10.40am:
Members present: Mr Iain Wright (Chair), Paul Blomfield, Peter Kyle, Amanda Milling, Amanda Solloway, Jo Stevens, Kelly Tolhurst, Craig Tracey, Chris White
Questions 64-130
Witnesses: Megan Dunn, President, National Union of Students, Sally Hunt, General Secretary, University and College Union, Professor Jon Scott, Pro Vice‑Chancellor (Student Experience), University of Leicester, and Stuart Cannell, Student Reviewer with the Quality Assurance Agency, gave evidence.
Q64 Chair: Good morning, ladies and gentlemen. Thank you for attending the Committee; we are very grateful for your time. For our purposes and for the record, could you tell us who you are and which organisation you are representing?
Megan Dunn: My name is Megan Dunn. I am the President of the National Union of Students.
Sally Hunt: My name is Sally Hunt. I am the General Secretary of the University and College Union.
Professor Scott: My name is Jon Scott. I am Pro Vice‑Chancellor (Student Experience), University of Leicester.
Stuart Cannell: My name is Stuart Cannell. I am a QAA student reviewer.
Q65 Chair: Professor Scott, may I start with you? What is the problem that the TEF and the Green Paper are trying to solve?
Professor Scott: That is a very good question because it implies that there is something broken in the university sector, which I think I would challenge.
Q66 Chair: Is the Minister wrong to suggest that there is patchy provision, or is he right to focus on prioritising teaching excellence?
Professor Scott: Teaching excellence, clearly, is something that we are all aiming for. Within any system with a large structure, not everybody is going to be at the peak of their game 100% of the time, so it depends how you define “patchy”. If you mean not everybody is absolutely excellent all the time, then that is true for every organisation in the country. It depends where you put that level. The UK brand for higher education is extremely strong and it is supported by the QAA, by the work of NUS and by the partnerships that we have. The implication that there is something broken with that system could be challenged.
Q67 Chair: Megan, may I turn to you? You represent students experiencing the quality of teaching every single day. Is the system broken?
Megan Dunn: It is fair to say that the debate around the quality in higher education is one of the most important for students—and that is quality and teaching quality. It is absolutely right that we should be talking about it and ensuring that we get it right, but there are real concerns about the timescale metrics proposed for TEF and that they are not reasonable or effective. That is exactly why we should take a bit more time to look at what is expected of TEF, to examine exactly what teaching quality looks like and to find a way of looking at teaching quality that is not simply about a quick way to raise tuition fees.
Q68 Chair: Sally, what is the experience of teachers and lecturers within the HE sector?
Sally Hunt: Their experience is that the system is broken, but it is not broken for the reasons that are being put out there. I do not think it was helpful to have a Minister use the word “lamentable”. That was highly inaccurate and it undermined the confidence of a lot of people who work very hard, particularly when you look at the student survey results that say over 80% of students are very satisfied with what is taking place in terms of the classroom.
What is broken is the way that the university sector is undertaking teaching, and he is right to look at that. What is taking place there is an exercise in insecure employment that then impacts on the quality that students have. That is a deliberate strategy from universities. If this starts to unpick or put a spotlight on that, it has to be a good thing. The questions have to be the right ones, not just statements saying, “This is about us having a go at the sector,” because the sector is very successful and the people who work in it are very good at what they do and work hard.
Q69 Chair: The Committee would like to explore the nature of employment within HE in a moment. Stuart, do you have anything to add in terms of whether the system is broken?
Stuart Cannell: I would say no as well. From what I have seen, the national standard is very high. Although not everyone is at 100% of their game all the time, everything is still relatively high in comparison.
Q70 Jo Stevens: Sally, I wanted to ask you about this point regarding the nature of insecure employment within the sector. We have seen a trend towards casualisation, temporary contracts, zero‑hours contracts. If that continues, how do you see that fitting within this teaching excellence framework and what is the impact on that insecure employment on how it will operate?
Sally Hunt: I will give you a bit of context. We are facing an exercise that proposes we get rid of the ability to have freedom of information. One of the things we found out using FOI, which we have shared with the Committee in our evidence, is that there are circa 200,000 teaching‑only contracts across the sector. Of that, around 100,000, 50%, are on fixed‑term, of which some are hourly rate, some are termly, some are atypical contracts. At the moment, the sector is heavily dependent on very insecure employment terms and conditions. If I put that against what you get in terms of the wider lay of the market, the ONS gives figures of around 6%. I am sitting looking at a sector that say they have to function—because that is what they will tell you when they come and talk to you—with insecure employment. That is 50% set against a wider 6% in the market. I might not be the brightest graduate in the world, but I would question that.
What that means is that, in terms of the quality that students have to deal with, they have a number of barriers before they even get into the room with the teacher. The teacher often is not given the secure terms and conditions that allow them to do their job well, by which I mean: not on scale and not therefore allowed to progress; not necessarily having access to an office, for example; not necessarily having access to the equipment that they need; and not being part of the structure of a university. What that means for the student is that the preparation needed is often very difficult to do and, going into the classroom, they are not getting the lesson that they should get. In terms of contact time, because someone on a fixed‑term contract is often jumping between contracts, they are not able to spend time doing the pastoral work and the backup work. They are often not able to form relationships.
That means, while I really do think Jo Johnson has asked the right questions in terms of TEF and how we do this, it is not possible with the current system for the people whom I represent to deliver back what the Government want. You have to have the debate about how people are employed in order to deliver higher education if you want to solve the problems of quality. Without that, this is a very interesting debate, but it is not one that will change what is happening.
Jo Stevens: Megan, do you have observations on that particular point?
Megan Dunn: We sometimes overlook the nature of postgraduate‑taught students when we talk about casualised workforces. A lot of postgraduate students teach, and they are very, very likely to be on casualised hours. Their working conditions are increasingly pressurised. We have done a lot of work, alongside UCU, on improving protections for postgraduates who teach, including a postgraduate employment charter. We would absolutely agree that looking at the casualised nature of university staff is very high up on the agenda for us.
Q71 Chair: Can I push you on that, Megan, because that is a very important point? Are you suggesting that there is a link between casualisation of teachers and lecturers within the HE sector and the quality of teaching that is provided for students?
Megan Dunn: There is absolutely a link between casualised labour and the amount of focus that staff are able to give teaching. We have to talk about the nature of the casualised workforce within the higher education sector and exactly how that impacts on where it is that staff are able to spend their time. Are they spending their time looking for the next contract because that is what they have to do, or are they spending their time building the relationships with students that Sally talks about and doing the pastoral work in order to ensure that they know the progress of their students, where they are up to and how it is that they can best improve?
Q72 Peter Kyle: On that point, I have been a postgraduate student and I have been a postgraduate doctoral student who has taught, so I have flashbacks when you are describing what it is like to suddenly be thrust at the very last moment into a room full of students who need tutoring, having been given very little preparation for it. The flip side is, if we are going to move to a system where postgraduates teach, are you saying that postgraduates need to be instructed and trained in teaching before going into the seminar room or doing the odd lecture on what research they are doing? Should a three‑year postgraduate course or period of studies have teaching included in it? Is that how you would like to go?
Megan Dunn: There are a number of different ways you could do it. I would start off by looking at how postgraduates who teach are paid. We find that lots of postgraduate students who are teaching are paid simply for their teaching hours and not for any of the preparation work that needs to be done. If you are looking at a system where that is how you pay postgraduate students, it is unsurprising if they have to pick up as many hours as possible in order to earn enough money to get by. Therefore, the focus cannot be placed on the preparation time, which would aid that time in the classroom so much more.
Q73 Peter Kyle: The discussion has got to a very interesting point, because a university is a place of learning and teaching, and the nature of postgraduate studies means that the lines are blurred. It is quite difficult to see how you would formalise a way forward in that, because you do not want to affect the free exchange of knowledge. Postgraduates who have something burning to say should be able to explore it with undergraduates, and undergraduates should have exposure to postgraduates. Also, some of us have been undergraduates exposed to terrible teaching by postgraduates and vice versa, looking at it the other way around. You want to get it right for both.
Megan Dunn: There is a huge value in peer learning and we should never underestimate that or try to take it away. We know from students across the board that they genuinely enjoy and find learning from their peers valuable, but we have to look at how that can happen in a way that means the teaching is still of good quality. Some of that will be looking at what it means to be a member of staff in an institution and what that means in terms of maternity pay, holiday pay and all of those sorts of things. If you are on a term‑time‑only contract or you are paid only for your teaching hours and not your preparation hours, it is really easy to see how the work cannot be done beforehand, through no fault of the postgraduate in question. Actually, that does impact on the quality of teaching. It is interesting that the TEF does not pick that up at all either.
Peter Kyle: I think Sally wants to speak.
Sally Hunt: I am just listening to what you are saying and thinking that it is not about saying one size fits all here. The whole joy of university education is that there is difference and there is room for that difference, because that is important. There are basic things that you can do. If you know that you have low morale because you have insecure employment, then you deal with the insecurity. If you know that you have issues in terms of confidence, then you put in place some form of continuing professional development. That does not have to be the same for everybody, but it can have a recognition of peer review or of support and mentoring. You can put in place structures that say, “We will support you in getting a training and teaching qualification.”
The divide that sits between research and teaching has always been one in which one is the lesser partner. If you want to raise teaching up, then make it something that has a qualification and a career path, and have those as clear and formal structures. I do not think it is difficult; I just think they have not done it because it is cheaper for them not to do it. People like you will always say yes; you are not in a position often to say no, because you want to progress, you want to do your research or, bless you, you might actually want to go in and try to teach, even if you are not very good at it.
That cannot be a system that is not possible to fix. I do not think it is hard. It is just, as I say, I do not think they have bothered doing it because they have not been called to account. Feel free to ask them, because I am absolutely staggered that we sit with 50% of our teaching staff on zero or fixed‑term contracts. It is extraordinary.
Professor Scott: To pick up on your very first point, I think things have moved on somewhat since your day—in the nicest possible way.
Peter Kyle: I was a mature student when I went to university.
Professor Scott: For example, in my institution—indeed, I am a QAA reviewer, so I have looked at lots of other institutions—all postgraduates who teach have to have undertaken training before they do that. We monitor their teaching and they are supported in what they do. The levels of payment vary, and I accept the comments about casualisation. But, in my area of biosciences, for example, if a student is demonstrating in a practical class, then they would get very little pay for preparation because most of it is doing a practical that they have done. If they are giving a lecture or something like that, then we would pay them a rate that included preparation time for the development.
Q74 Peter Kyle: What is the nature of the training? You say you have teacher training.
Professor Scott: We have a learning institute or professional development unit that provides training for everybody who teaches. That applies to postgraduates who teach; it applies to postdocs; it applies to lecturers as well. New lecturers who come in have to undertake training and teaching. If you are coming in as a lecturer, you would be expected to take our postgraduate certification in higher education and be working towards one of the HEA qualifications. Indeed, we expect our graduate teachers to be going for associate fellowship of the HEA. We build that in all the way through.
Sally Hunt: That is good; it just does not fit what my postbag tells me. My postbag, electronic though it is now, has story after story of people who do not have that experience. What you are outlining is the right way to do it and it is a good way to do it, but the truth of the matter is that there are thousands of people out there who are inadequately supported and are not in a position where they are able to assert what they need. They are not able to articulate that well, given where they are in the employment chain. That is something that is common. It is still common throughout the university system and it is impacting in terms of how this whole debate will have any fruition.
If the Department wants to get quality education and teaching, and it wants to do that through TEF, it has to deal with what is happening in terms of insecurity of employment and in terms of how that sits outside a lot of the structures. Otherwise, the people Megan represents are going to be at the sharp end of this, being asked to pay higher fees linked with metrics for reasons that I still cannot work out, and we are not going to get what Jo Johnson wants. What he wants is good higher education teaching, and that is something we should all aspire to.
Q75 Paul Blomfield: I would like to move on to a different area and ask Stuart a couple of questions. Within the Green Paper, there is a focus on the NSS as a measure of student satisfaction. Now, you are engaged in providing student perspective to quality assurance in a qualitatively different way. What do you bring to the process that simply looking at the NSS stats does not?
Stuart Cannell: Evidence can only speak so much; it is just facts, really. One of the things about the HER process, which I think all the reviewers admire, is that, when you get to that review visit, it cements everything. You can look at statistics and you can read papers and what they say, but the best thing about the process is when you get to the institute and get a feel of the place, which conveys things that the NSS could not say on its own.
Q76 Paul Blomfield: You need a qualitative insight, not just numbers.
Stuart Cannell: Yes, I believe so.
Q77 Paul Blomfield: What have you seen from the visits you have been on to institutions? How many university visits have you been involved in?
Stuart Cannell: I have been involved in five so far.
Q78 Paul Blomfield: Presumably across a range of institutions.
Stuart Cannell: Yes, across a range of colleges and universities.
Q79 Paul Blomfield: What is your impression of the way that those institutions are approaching teaching quality?
Stuart Cannell: It varies. I would say that all the institutes I have seen so far have had a good standard, if not a high standard, of teaching. Some institutes approach the higher education academy fellowships; they would have that as a strategic objective. Some people want to improve their teaching through PGCerts or simply through sharing good practice. They might have mechanisms internally and periodic review to enable that. That varies from institute to institute, but it is something that is standard throughout.
Q80 Paul Blomfield: Have you any thoughts on the approach that the Government are suggesting with the teaching excellence framework in terms of the experience that you have gained from that process?
Stuart Cannell: It was good to see that the student would be involved in the assessment process of TEF. One aspect within the HER process that is worth mentioning is that you feel very much like a peer. Even though, technically, I am a student reviewer, I am a reviewer at the end of the day. I can get any section of the quality code to look at. I am treated as an equal. In terms of students being involved in the TEF assessment, I would like to see that maintained.
Q81 Paul Blomfield: Can I move on to Megan? I am sure that you agree with Stuart’s point in terms of the student involvement in the process. I wondered if you had any thoughts on that particular issue.
Megan Dunn: There are a number of questions, and the question about metrics is a really good place to start. The metrics act as an indicator for areas that need further exploration, but struggle to do much more than that. When we talk about the metrics we are looking at using for the teaching excellence framework, I have real concerns around whether they are indicators of teaching quality at all. Employment data, for instance, I do not feel is a suitable indicator of teaching quality.
In terms of students and their involvement, we have been told for a number of years now that students will be put at the heart of the system. That was, indeed, the line that was talked about in 2010 around the rise in tuition fees, particularly the link between teaching quality and tuition fees. We were told that, if students were to pay a higher amount, they would be put at the heart of the system and that teaching and quality would revolve around them more. Five years on, we are being told that, in order to put teaching quality and students at the heart of the system and to ensure that teaching is good quality, we need to raise tuition fees again. That is a really interesting link to explore.
Q82 Chair: You do not think that, in the last five years, we have seen an improvement in teaching quality, then.
Megan Dunn: I do not think there are any indicators that we have seen an improvement in teaching quality. What we have seen is a lot of discussion about setting up a system that is in students’ interests. For instance, the Green Paper talks about an office for students that is funded by the sector and institutions and regulated by the Government. It says that that office would determine students’ interests, but there is no indication as of yet that it would ever talk to a student.
On this idea that we are putting students at the heart of the system, for us at NUS, we would like to talk about a partnership between institutions, staff, academics and students in terms of how we create teaching quality that is relevant in context. You have to remember the huge diversity of higher education institutions, from colleges that provide higher education to postgraduate‑only institutions, from large generalist institutions to very small specialist institutions. What does teaching quality look like in that context for those students? A partnership approach is the only way that we can do that. That is supported throughout the sector. The sector funds a number of bodies, including the Student Engagement Partnership, which is co‑funded across the sector, to look at how we support partnership and partnership work.
Q83 Paul Blomfield: I would like to pursue your point about metrics a little further, Megan, in two areas. First, on employment data, we have heard a lot of witnesses tell us that that is a flawed metric. What is your specific concern about it?
Megan Dunn: For me, employment data is probably a proxy too far in terms of teaching excellence. Research from HEFCE in 2014 tracked 130,000 students and found that state‑school educated students are more likely to get a good degree than privately educated students. Yet the Sutton Trust indicates that privately educated graduates are, on average, likely to earn £4,500 more a year than their state school counterparts three years after leaving university. If you can pick out high‑earning graduates before they enter an institution, what does this mean in terms of recruitment strategies for institutions that want to pick only the highest earners?
It is also really interesting to note the gender pay gap and the fact that, from mid‑November, women are working for free for the rest of the year. What does that mean in terms of whom we recruit and why, particularly in light of what we know around BME graduates and their employment prospects as well? There are some real concerns: not only that employment data does not serve as a proxy for teaching excellence, but that it can drive really perverse incentives around recruitment for institutions who, because of all the incentives that are put into the teaching excellence framework, want to be seen to be doing well.
Q84 Paul Blomfield: That is helpful and reflects the views we have heard from others. What about NSS? It could be argued that NSS has provided some focus on teaching quality. Would you agree that universities upped their game on assessment and feedback in response to poor NSS ratings? Do you have concerns about that as a metric?
Megan Dunn: The NSS often acts as a place to look for more concerns. Students often say, “My feedback and assessment is not how I would like it to be” or “My course organisation and management is not how I would like it to be”, but there is really no data other than whether they think it is good or not. There is no indication of why or whether it was on the institution’s end of things. For places that have partnerships with other organisations, if their building is part owned by their institution and part by somebody else, what does that mean in terms of who that is down to? It acts by saying, “Here is an area where we should focus and look deeper into what happens”, but at no point do we ask what students care about.
We say, “How is your feedback and assessment?” or “How is your course organisation and management?” No one ever says, “What is it about your course that has made your teaching good or not good?” We never ask that question of students. We provide them with the parameters within which they can answer. Part of this recognition of what teaching quality is is that it will be different for different students. We have to start talking about what students care about, because excellent teaching only happens if there is excellent learning taking place. We cannot provide such strict parameters to students for the terms in which they can talk about their teaching.
At present, we do not have any system whereby we ask students what it is that they care about and how their teaching has been conducted. Simply looking at metrics that have been derived from questions students have never come up with themselves is probably not a particularly good proxy for teaching excellence either. There has to be a better way of doing that.
Q85 Chris White: This is to you, Megan. You say that the employment data is a metric or a proxy too far. I can see some of your arguing point behind that. You answered my question a bit in what you just said, but I would be interested to know if you had a template of what you think would be a good idea for students to respond to in terms of their views on the level of quality of teaching. Also, you mention that you feel—I may be wrong in what I think you said—that the quality of teaching has not improved over the last five years.
Megan Dunn: There has been no notable driver from tuition fees that has indicated that.
Q86 Chris White: Do you have data to support that?
Megan Dunn: I will just double check what is on here. If not, we can send through some of the data that shows no particular change in student satisfaction with their teaching over the last five years. We have to talk about where tuition fee spending is going. There was an expectation from students, rightly or wrongly, that increased tuition fees would mean increased funding for institutions, which would mean that the quality of their teaching would improve. Actually, because it was replacing public funds in that way, there has been no additional funding for universities to put into teaching.
Increased marketisation and competition between institutions means that institutions have to spend far more time looking at the recruitment of students. Unfortunately, often, the focus on quality and improving teaching for students is a bullet point somewhere in a strategy rather than the focus of the institution, because the institution’s focus has to be recruitment at this point, particularly since the lifting of the student number controls.
Sally Hunt: Let us remember that the NSS is saying that 86% are satisfied, which I think most MPs would be quite happy about. In terms of the five‑year improvement, if you are sitting at 86%, we need to be very clear that we are not talking here about people who are doing their job badly. They are doing their job well. The challenge there is how you use the instruments to analyse what could be better.
We famously do not like the student survey because we think it is a blunt instrument and it does not necessarily ask the right questions, in the way that Megan has talked about. It is definitely something that institutions use, as you were asking, because why wouldn’t you? But that is not the same as saying it helps the teacher and the student work out something better, because that is a different conversation. That is not going to be solved by making it even blunter by the use of metrics. All you are going to do is remove the peer group element, which is about getting under the skin, for both staff and students, in such a way that you really see the improvements and people are able to develop and learn.
That is not what you get from surveys. Stats will tell you all sorts of things, but, in terms of what it is possible to change, if the metrics line is where people want to go, then they have to say, “Does the suggest that metrics will show you what quality is?” I would suggest, and we have given you this in our evidence, that the King’s College survey shows it that does exactly the opposite of that. Where we had 13 institutions that were failing, it did not show any of those as failures when it just used metrics. You have to look at that, because Megan is right in saying that students have the right to walk into that room and have a good teaching experience. They have to have methodologies that support them in that. Do not make them the customer who is always right, because, frankly, an undergraduate isn’t always. There is a much broader conversation to be had there in terms of what is actually good teaching and what is not.
Q87 Chris White: I have one last question. That template would be interesting, if there is something you would like to see as a model. What would you say in terms of the quality of teaching being patchy? Forgive me for bringing up Warwick University, which went from 60th in the world last year to around 40th in the world this year. Apart from our usual annual surveys, how would you know that that was the quality of teaching that you would expect from that institution?
Sally Hunt: I will take it away from Warwick because you will understand that, as the General Secretary of a union that represents Warwick plus all sorts of others, I will not say that one is awful, one is good and one is average.
Chris White: I have to say that they are outstanding.
Sally Hunt: That is different. I think all the people I represent do a very, very good job, as a matter of record. You have to recognise, with all of these league tables, that they are a blunt instrument. Any parent who is working with a 16 or 17 year‑old, trying to make choices, is looking at things that are very surface‑level in terms of what they are able to find out about. What you actually need to know about an institution is what is going on in terms of the number of students who are having contact time with their tutor and their ability to engage and work with them.
That means things like: is the teacher actually going to be there and available for them? Are they going to have the opportunity to have those one‑to‑one conversations? Are they going to be able to say, “I want to sit in smaller groups and talk and work with them”, rather than what I would call sausage factories, which is what is happening with a lot of students now in terms of contact being purely through a lecture rather than anything else? You are never going to get that through those figures. We would like to see—and I do not think it is hard for us to do it—something that says, “Take peer review and respect that” or, for example, “Strengthen what is happening in terms of external examiners.” Within that, you can add metrics that support it, but you have to have the right metrics.
From my very selfish point of view—and I am absolutely clear about this—I would like to see within that a clear question: “How do you employ your staff?”, because I think there is a direct link. If you have well paid, well supported, secure staff, they will be able to give their time to those students, and none of the tests you have at the moment enable us to ask that question. Until you do, you are not going to get what you need and Megan is constantly going to be asking, “Why, why, why are you saying that students matter when we are not seeing change?”, because you will not.
Q88 Amanda Milling: Some of the questions have just been answered. Megan, just so I am clear on this, you are saying that student input is good, but the NSS currently is not really fit for purpose.
Megan Dunn: I think you would hear concerns across the sector around not simply the NSS in itself, because it does provide useful indicators of where there are areas to focus on, but the amount of credence we give what is simply a tool to look for areas for further observation. To use it as the tool that tells you whether students like their institution or not is probably flawed, and I think you would hear that echoed across the sector.
Q89 Amanda Milling: But you could argue that tweaks could be made to it or it could be evolved to be able to deliver further granularity.
Megan Dunn: We are further evolving the NSS, and the NUS is and will be involved in that process, but using it as the be all and end all is flawed.
Q90 Amanda Milling: Because this is a basket of metrics, Sally, if there was an equivalent to an employee engagement metric in there, do you think that might help?
Sally Hunt: What do you mean by engagement?
Q91 Amanda Milling: Basically, in the private sector, in lots of organisations, they will have a basket of metrics, which might include customer satisfaction, advocacy and a whole host of things. Also included in that is how engaged an employee is as well. It is quite complicated to explain, but I suppose I am asking you this. You have talked about the contracts. I just wonder, if you had something with feedback from lecturers or those people working in the sector, whether that would help to address your concerns?
Sally Hunt: You have to be aware that they are excluded from this whole process at this point in time. There is no ability for them to engage with the Department or HEFCE directly. We, as a trade union, give them voice through our collective negotiations, but often we find that, when ask questions about these most insecure members of staff, they are the ones who are excluded from the terms and conditions that we negotiate nationally. My point would be: on the surface, yes, more engagement, but it has to be the right sort of engagement. They have to be able to be represented as part of the community of academics.
In other words, where I am going in and negotiating for one group of members who are on permanent contracts, that should reflect through in terms of those who are coming into their careers. Those who are starting their careers should not feel like second‑class citizens. I am genuinely worried—I do not think anyone else is saying it, so I am going to say it again—about this issue of removing freedom of information as a method we are able to use to call universities to account. It is something that really worries me. Only through that method have we been able to uncover what is taking place for some of these most insecure people.
If we have to have engagement, yes, and if it becomes a metric, I would be very pleased. I would, though, like to see good clear negotiating agreements across the sectors that are open to people in terms of transparency. I would like to understand why that is not taking place at the moment. You cannot simply just say it would be useful to have them engaged; it has to be “how?”
Q92 Craig Tracey: I want to follow up on a point Chris made and come back to you, Megan, again. Jon might like to follow up on it. You are talking about data on teaching standards over the last five years. Presumably that is from a student perspective. We are saying there has been no noticeable change. Do you not think it is also fair to say that student satisfaction is not always the same as teaching quality?
Megan Dunn: Absolutely. That is probably one of the reasons why there are questions around whether the NSS is appropriate to use in a teaching excellence framework. Potentially, one of the biggest problems with the metrics that are talked about in terms of the teaching excellence framework is that they are not, in fact, metrics for teaching excellence; they are not robustly designed to talk about behaviours around teaching excellence. We do not know what behaviours they will encourage and what perverse incentives they will put in place.
The time frame is one of our biggest concerns with this. The metrics used are, at best, proxies for teaching excellence and, at worst, almost entirely removed, and we do not know what behaviours they will encourage in institutions. Institutions have been fairly open about the drive that they feel to game metrics, as such, and to talk about how they can look like they are improving in certain metrics and what they can do around that. Until we understand what metrics we are using and how they will encourage institutions’ behaviours, it is fairly irresponsible for the TEF to go forward.
Craig Tracey: Do you have a university perspective on that?
Professor Scott: I have to agree with quite a lot of that. If you do the stats and take the dreaded question 22, which is the student satisfaction one that feeds into all the league tables, that does not correlate, for example, with the questions on teaching or on assessment feedback. The closest correlation is around the organisation of the course—in other words, were the lectures at a time that was convenient? Looking to other countries—and we have been trying to drive this through—if you are going to have to use metrics, there are much more sophisticated metrics, for example, in the States and Australia. You have the NSSE in the States, which is broader based. It is around student engagement, so it is a two‑way process. If you have to use metrics, we can develop more sophisticated ones and we need to. There is a very clear picture there.
The metrics that are identified—again, I would pick up on the DLHE data—are in no way a proxy for the quality of the teaching. Whether someone has got a graduate job six months post‑graduation does not tell you anything about the quality of the teaching they have had. We need to be much more sophisticated and to think about how we are going to do this. The type of quality assessment framework that QAA has does provide some of that benchmarking and, going back to what I said right at the beginning, it provides the kitemark that sets UK HE on an international market. We recruit a large number of international students. In fact, I would go so far as to say that universities like mine are dependent on them, and that comes through because of that kitemark that we have there. If we jeopardise that as well, then the whole sector is going to be in danger.
Q93 Chair: Megan, and perhaps Stuart, can I take you down a road that you have travelled with Paul, Chris and Craig? If we put students at the very heart of this framework, what are the characteristics of a good teaching excellence framework? What do you want to see as part of a teaching excellence framework?
Megan Dunn: First, I should say that teaching excellence is absolutely an appropriate focus and one that we should be looking at. For us, teaching excellence consists of five main aspects and principles. One of those is that excellent teaching happens at a disciplinary level, so an institution‑wide look at teaching is probably not ideal. Excellent teaching happens at a disciplinary level, but there is a responsibility on institutions to create an environment in which excellent teaching can thrive. Excellent teaching has to be excellent for all students and not just some, because excellent teaching is inclusive and enables all students to learn. With data around attainment gaps and things like that, we see that some students are entering the classroom and finding it more difficult to learn than others. That is something we need to be looking at.
Q94 Chair: On that basis, should TEF include metrics about widening participation?
Megan Dunn: Again, it is around contextualising those and ensuring that we have tested what behaviours they encourage.
Sally Hunt: And the link to fees.
Megan Dunn: Widening participation should be a focus, but Sally is right about the link to fees. If we simply say, “If you are better at widening participation, then your fees go up”, I think lots of people would understand that there is a very strange interrelation between those things. There are a number of things that excellent teaching looks like.
Saying to students that you have to pay more to get better teaching is an issue that we have not touched on yet. We were told that you would pay £9,000 and get excellent teaching. We are now being told that, if you want to get excellent teaching, you have to pay more than the institution down the road. If we are saying to students that you can pay £9,000 and not get excellent teaching, then there is a problem. More than anything else, if we are saying we can send our students into higher education, whether a college or a university, and they do not get excellent teaching, I would question what the point of university is at all. A link to tuition fees is dangerous.
Q95 Chair: You think that we should decouple teaching excellence from the ability to raise tuition fees.
Megan Dunn: Absolutely.
Chair: Stuart, do you have anything to add?
Stuart Cannell: I would like to second that and also add in about TEF at an institution level. It is unfair for students, in years to come, if they go to an institute with a level 4 TEF, they go on a course and the teaching is terrible but that has slipped through the cracks. If there is an institute with a level 1 or 2 TEF but it has a course with fantastic teaching, as a student looking at those two options, you would immediately want to go to the level 4 TEF institute. TEF really needs to be at a departmental level and not at an institutional level.
Chair: Subject level?
Stuart Cannell: Yes, as low down as it can go, so it is as transparent as possible. I feel that, if an institute has a level 4 TEF and it is not deparmentalised or course‑based, that is a flag that they could wave to increase student numbers, and perhaps an unfair flag.
Q96 Paul Blomfield: I want to pursue with Megan the decoupling of fees and TEF. I can see why you would be opposed to any fees increase; the National Union of Students always has to be. Whatever we think of the current system, at some stage the fees cap will change. I am interested in exploring why you are particularly concerned that any change in fees should not be linked to the TEF.
Megan Dunn: I will first touch on the obvious statement. You are, of course, right: as the National Union of Students, we oppose the rise in tuition fees. When it comes to the TEF, we have to look at the issue of variable tuition fees and the incentives around them. If you genuinely believe that the TEF is an accurate reflection of teaching excellence, you will make it harder for institutions that are not teaching as well to get better as the gap in income of institutions grows. You are saying that poor performance should be penalised, if you believe they are poor performers and, therefore, the gap between them and the high‑performing institutions grows. It becomes harder for institutions to catch up to other institutions, and, worse than that, there will come a point where there is no indicator between fees and quality, even if you believe that fees indicate quality in the first place.
I have looked at some of this. If you increase tuition fees year on year, you do well in the TEF for the first six years and interest rates and inflation are good, you will end up with a tuition fee in the ballpark of £10,500. For the next two years, the institution could not pass the TEF and not have any right to raise tuition fees. If the institution down the road did not do well for the first six years but made massive leaps forward and for the last two years has been sector‑leading in their teaching excellence, but the inflation rate has not been so good, their fees might be at £9,200, because that is the amount they have been able to raise tuition fees by. To say to students that is an indicator of teaching quality is hugely flawed. It is not an indicator of quality at all. The link between tuition fees and the TEF is one that I absolutely believe should be decoupled.
Sally Hunt: Certain institutions have specialised or had a much better track record in bringing in non‑traditional students, who are the students least likely to have wanted to go to university comparative to their peers. If you say that the institutions that do well in TEF are able to raise their fees, those students will be the ones going to institutions with higher fees, which seems rather bizarre in terms of the equality of opportunity and the equality of price. Those least likely to have the money will go to those institutions that are possibly, by this link, able to raise their fees. It is a very skewed logic and it is very dangerous.
Professor Scott: I agree entirely that the coupling of the TEF with fees does not help the sector. Yes, sure, we would like more money; I am not arguing against us needing more money. One of the cases being made is that it forms an incentive to institutions to deliver better teaching. I would seriously challenge that, because, on the one hand, if you have a TEF 4, then you are going to be attractive to students. In a marketplace for students, a TEF 4 institution will be in a much better position to recruit students, because students will be attracted to that institution. In an environment where you are looking to increase a number of providers and the number of 18 year‑olds is dropping, then that competition is going to be a real driver for institutions. Going back to what I was saying about international students, if you have a TEF 4 versus a TEF 1, you are going to be the place that students from China will come to. I would argue that the rationale for linking those two together is flawed and the system will work to drive it anyway because we will want that TEF 4.
Q97 Paul Blomfield: That is very helpful. Stuart, do you have a view on the issue of linking fees?
Stuart Cannell: There is one thing that I would like to pick up on. During the higher education review process, we look in detail at a lot of this; it is a very thorough process. There is a chapter dedicated to learning and teaching. A lot of good practice we would identify with during that process.
Chair: Thank you very much for your evidence; we are really grateful. Thank you very much for your time.
Examination of Witnesses
Witnesses: Professor James Wilsdon, University of Sussex, Matt Hiely‑Rayner, Head of Planning, Kingston University, John Gill, Editor, Times Higher Education, and Nicki Horseman, Lead HE Data Analyst, Times Higher Education, gave evidence.
Q98 Chair: Thank you for attending our session on assessing quality in higher education. Could you tell us who you are and which organisation you represent?
Professor Wilsdon: I am James Wilsdon. I am professor of science and democracy in the Science Policy Research Unit at the University of Sussex. I am here primarily because I chaired an independent review into the role of metrics in research management and assessment for HEFCE and BIS, which published earlier this year.
Matt Hiely‑Rayner: I am Matt Hiely‑Rayner. I am Head of Planning at Kingston University. I am here because I am also Director of Intelligent Metrix, an organisation that compiles statistics for The Guardian’s university guides.
John Gill: I am John Gill. I am editor of Times Higher Education magazine and publisher of the THE World University Rankings.
Nicki Horseman: I am Nicki Horseman. I am from Times Higher Education. I work on the world rankings.
Q99 Chair: One thing that strikes me is that, in this country, we have an incredibly diverse higher education sector. Is it possible, given that diversity, to have a common set of metrics?
Professor Wilsdon: Diversity is a real challenge with quantitative indicators. I should preface my remarks by saying that we looked in detail at the research side of the system rather than the teaching side, but a lot of the general principles about both the possibilities and the limitations of using quantitative indicators within the system apply. Diversity is one of those issues. If you have a system that has institutions with a variety of different missions and a diverse set of objectives that you are trying to achieve, designing frameworks for measuring excellence, whether it is research excellence through the REF or teaching excellence through the TEF, is a real challenge. It is not impossible, and a lot of work has gone on, certainly on the research side, over the last 20 years to try to grapple with this. On the teaching side, there are some indicators that work better than others, but, in toto, I would agree on many of the issues raised by the previous witnesses.
Q100 Chair: Can I push you on that, James, before I ask other witnesses about the common set of metrics and diversity? Given that we want to help the Minister provide an excellent teaching excellence framework, what lessons can we take from the REF in terms of those difficulties in establishing a common set of metrics?
Professor Wilsdon: The REF, as I am sure Committee Members will be aware, is primarily a peer review‑led process. Within the REF, there are 36 individual panels, which can decide whether or not to draw on quantitative data in their deliberations, but the idea is that, even when they do, that data is a support to their expert judgment, not the basis on which they make the judgment. There is now, within the Green Paper, a separate section, section D, consulting on whether that system should be overturned and whether we should have an entirely metrics‑based REF, which takes us further in that direction.
In terms of general points, one thing that one can say about the REF, looking at the successive cycles of the RAE and the REF since 1986, when the RAE was introduced, is that you have had experimentation, testing and piloting at every stage of significant change. One of the concerns on the TEF side right now, which other witnesses have raised this morning, is that, whatever one thinks about the objectives of the exercise, the sheer speed of implementation, even with this initial TEF‑lite mode, raises concerns in respect of our capacity to test and assess the range of behaviours that get incentivised by different metrics when you introduce them. That is one big factor.
The other broad thing is that we set out in our Metric Tide report a framework for thinking about responsible metrics, which is a phrase we use to try to convey the fact that metrics in and of themselves are neither good nor bad. They are tools; they are indicators. They can be used sensibly, appropriately, responsibly in ways that support and respect diversity and all the other qualities you want to see in the system, or they can be used in ways that erode those qualities.
On the research side, we certainly have significant evidence of the perverse consequences that can arise when you place too much emphasis on particular quantitative indicators in terms of the way that has affected research culture and behaviours. Whatever we now do on the teaching side, it is important that we try to join these two conversations up and ensure that we are at the very least learning the lessons from the research side in its introduction.
Q101 Chair: We might come on to that in a moment. To the other witnesses, can we have a common set of metrics, given the huge diversity?
Matt Hiely‑Rayner: Yes, I believe we can. There are some indicators. It is important to respect that there is diversity within institutions as well as between institutions. Some metrics represent universal truths, where there is a value judgment: a certain outcome for a student should always be regarded as good; other outcomes should always be regarded as bad. Then there are some metrics that need to be more nuanced. For example, value‑added scores try to take into account where a student is starting from rather than just valuing student achievement on an absolute scale, and I think some metrics have to be regarded in that sense.
John Gill: It is problematic, particularly for teaching. Our rankings look at the world, but all we are really looking at is the top 4% of research‑intensive institutions around the world. They are, in a way, already a fairly homogenous group. These are institutions that are essentially research‑led; they are focused on things that we can measure very clearly. When you come to a national system, suddenly you have the full spectrum. It is a much more diverse environment, as you say, and that is something we absolutely should be looking to preserve. The danger with a common set of metrics in a national system is that you risk leading people, perhaps unwittingly, down the path to homogeneity, which I think is a bad thing.
Nicki Horseman: I agree it is problematic. It is really important to understand what you are trying to measure. There has been a lot of discussion between educational gain and education performance. Rankings tend to measure educational performance and not education gain. If you think about reputation surveys, it is about how universities are performing overall, not the distance travelled. It needs a lot of care to come up with something that could address the diversity of the sector.
Q102 Chair: John, I understand your point in terms of your world rankings. You look across the globe. It might be a relatively homogenous group, but are there any lessons that the Minister can take from what you are doing in terms of those rankings? Of the five areas, one is teaching and the learning environment. What lessons can we learn from what you do?
John Gill: The distinction that it is really important to make is that we are looking at the learning environment. That is not the same as teaching quality or teaching excellence; call it what you like. We take a Humboldtian view of what the learning environment should look like within a leading research‑led institution. We are able to look at things like the number of doctorates awarded per academic staff and things that relate to the business of being a research intensive university, as well as qualitative measures like reputation, which are really important. It is easy to dismiss qualitative measures and say that they are subjective. Of course reputation is subjective, but it matters. It matter to students, to consumers.
We have put together a basket of metrics that we think builds a picture of a learning environment. To measure teaching quality at a subject level or even down to an individual academic is much harder to do. It is the holy grail of the rankings game. We are very proud of being the only world ranking that has this teaching environment score within it. We think it is right that we build a rounded view of what a university is and we should not just focus on research, as some other ranking systems do. I would be careful about equating it to teaching quality. I am not sure that the metrics that have been picked for TEF in the first instance do equate to teaching quality. I think they have been chosen because they are pretty much the only metrics that could have been chosen.
You can only measure what is measureable. That is one of the lessons we would take from rankings, and I have yet to see evidence that teaching quality is really measurable in a true sense. One of the problems with the metrics that have been selected here is that they are outcome measures, so we are looking at things like employability and graduate salaries. What are you actually measuring there? You are measuring the type of student that a university has, really, not what the university has done with that student. If you are coming from Eton straight into PPE at Oxford and straight into the Cabinet, then good for you.
Chair: Could you narrow it down a bit more, please?
John Gill: That is not something we would measure and say it was because of the brilliant teaching that they went from A to B. There are big issues there about what the metrics are. What other metrics are available to the Minister, at the moment, for TEF? I do not think there are that many other choices. That is not to say we could not develop more credible metrics, but at the moment we are not quite there.
Q103 Chair: Just to carry on down this path, and I am putting words in your mouth, John, are you suggesting that the Minister is wrong in his priority? While he is right in what he wants to achieve, rather than focusing on teaching excellence, he should be thinking of a learning environment excellence framework—not a TEF; more like a LEEF. Is that fair?
John Gill: It is a commonly held view among vice‑chancellors that the learning environment would be a better measure, not least because, as I have heard it said, though “learning” or “teaching” is semantics, really, if you call it teaching you will end up focusing on the teacher, and five years down the line we will have a REF‑style transfer market for teachers who are the latest must‑have.
Q104 Chair: Is that a bad thing?
John Gill: I do not think it is necessarily a bad thing. As a magazine, THE covers, week in week out, the fact that teaching is an undervalued activity within universities. It is undoubtedly the case. It is easy for people to say that it is not true, but I can give you one example. Where does the prestige of universities come from? It tends to come from their research stars, because that is what has been valued; that has what the REF has required universities to value; that is where the money has flowed. The result is that students are choosing universities on the strength of research quality. They are then turning up and perhaps finding that the research stars are not teaching them and do not want to teach them.
When MOOCs were all the rage a couple of years ago, when suddenly this was the most exciting thing to happen in higher education and MOOCs were the future, I had a vice‑chancellor telling me that one of his research stars, who had never shown the slightest interest in teaching, was banging on his door saying, “I want my MOOC.” This was the first time he had ever wanted to teach, because it was exciting and new. I do not think it is true that there is not a problem within institutions that needs to be addressed. The Minister is right to be focusing on this. Whether the basket of metrics that has been selected, because they are the only metrics that can be selected at this stage, is going to get the job done I am not convinced.
Q105 Paul Blomfield: I wonder if I could follow up, John, on specifically that point. You say—and I think we all agree, and the sector seems to—that the Minister is right to focus on teaching quality, and there seems to be general concern about the metrics. The thing that we are keen to explore is, if not this, then what?
John Gill: Learning environment is one option, but that might not be the right option either. Student engagement is a very obvious place to go; the NSS is already starting to move down that route slightly. Graham Gibbs, formerly of the Oxford Learning Institute, wrote a series of articles for Times Higher, which I commend to you, about what we could measure and what good teaching looks like. He is very keen on an equivalent of the NSSE in the US, which asks questions such as, “To what extent do you engage with your faculty outside of class? How much do you contribute in class?” and things like this that are not just, “Are you satisfied?”
The problem with satisfaction, which we have written about in Times Higher, is the phenomenon called yea‑saying, where students are perhaps pressured to give good NSS scores, not overtly by their institutions, but because it now forms an important part of a university’s brand to have a really high NSS score. Students understand that the university’s brand equates to the job market, careers and salaries further down the line. There are lots of things that can go wrong with some of these scores. Student engagement is one, I would suggest.
One of the obvious ones would be teaching qualifications, perhaps. There are dangers there too. We had a news article and a research study recently that said the problem with teaching qualifications is you get much more homogenous teaching. You will have teachers who will get a large number of students up to a pass mark, but you will not have teachers who get a smaller number of students to really brilliant levels, who really spot the stars. Maybe that is a good thing, but it is something that needs to be considered.
Another obvious one that everyone has talked about is learning gain, value added and how we measure that. It was interesting that there was an OECD project to try to do this. The Assessment of Higher Education Learning Outcomes, or AHELO, project was a serious if flawed attempt to do a piece of work around this that would have resulted in international comparable data, and the UK opted out of doing it. There may be a question there about universities being a little scared; there are risks involved. UK HE has an incredibly strong brand globally, and there are dangers that, if you start to measure this, some institutions may not be as good as we all think they are at that aspect of what they should be doing.
Q106 Paul Blomfield: Can I press specifically on learning gain? Coming at this to a degree from the outside, it is the obvious thing to be looking at, and yet how would we—not only in your view, but I would be interested in hearing from the rest of the panel—effectively begin to measure learning gain. In the States, for example, we have heard there are entry tests and exit tests, but that approach has its unintended consequences too, I guess.
Nicki Horseman: It is certainly not an easy thing to do. You can look at it across a number of areas. One is about whether you measure it at the start or at the finish. Clearly, we have problems around degree classifications and equivalents, so what is your exit measure? Is it employment? We have talked about how social background is going to have a big influence there. HESA provides benchmark data against some of its performance indicators. Based on relevant factors like, for a university, the entry qualifications of its intake, the subjects that they have studied and the age profile, it produces benchmarks.
If you look at that, the types of institutions that are exceeding the benchmark performance are not the same ones you would look at for absolute performance. It is about measuring different things and asking what you are really trying to get to. For the institutions that already excel because they are getting the elite students, there is not very far they can take them in terms of learning gain necessarily. You would hope that they are not taking them backwards, but they do not have the same ability to improve.
Q107 Paul Blomfield: I was going to ask Matt and James to respond to the same question. If not this, then what, and how do we measure learning gain?
Matt Hiely‑Rayner: We use a value‑added score in The Guardian’s guide, which gives each student a probability of achieving a good degree. It works; we wanted to have a sense of student achievement in our guide, but there are rumours of it promoting grade inflation. A league table does not exert the same influence that the Government recognising teaching excellence would, so it could be problematic to start recognising excellence by using a measure like value‑added, because it would be in the gift of autonomous institutions to award more firsts and 2:1s.
Professor Wilsdon: I would agree with that. To the credit of BIS, they do recognise in the Green Paper that a range of proxy indicators is required. Some of those proxies currently do not exist so, as John said, the natural thing to do is reach for the measures that we do have. To come back to the stuff that Megan was saying previously, the real problem arises when you link it to the funding system, as you introduce very powerful levers for strategic responses of various kinds, whether that is grade inflation or other behavioural changes.
Sometimes, with quantitative indicators, you want to introduce behavioural changes. The impact element of the REF many think has been a very positive additional element to the research system; it has encouraged people to think externally. But, with some of the measures proposed for the TEF, the concern that many in the sector have is that we have not thought through, cumulatively, what new incentives are going to be introduced to the system, even if the measures are being adopted with the best of intentions.
Q108 Chris White: John, I suppose you would see your rankings as pretty much a gold standard. That would be fair to say; we are all interested in how they come out each year. I have a couple of questions. Do you ever get challenged by the universities that they think you are totally wrong and it is unfair? Do you ever think of changing the criteria that you use, year on year? There is a question I would like to ask the entire panel, if that is okay. We have heard this morning that the quality of teaching has not improved over the last five years, but each one of you then goes on to say that we have an incredibly strong international brand. How do we relate those two statements?
John Gill: On the international brand, that is because the brand is largely focused on research strength. We have an incredibly strong research sector, and that tends to be where the prestige is sat. That is really what Jo Johnson is trying to tackle with the TEF. In terms of whether we get challenged, of course we do. We are looking at the top 4% of research‑intensive institutions in our top 800. There are lots of universities that will never fit that profile and would like us to be measuring things that would perhaps make them look better, some of them incredibly good institutions. Take a leading liberal arts college in the US: you would not argue that they are not a spectacular institution doing brilliant things around teaching, but, because they are not the research power that we are looking at, they will not be as high up the rankings tables as they would like to be and some would argue should be.
One of the areas that I have been lobbied on by UK vice‑chancellors has been learning gain. People say that would be a real game changer, and quality and the actual business of what the institution is doing with students would surface. For the reasons we have been discussing, that is very hard to do.
The other thing that I would say is that we are looking at the global sector, if you can call it a sector globally, and the result is that we are limited in terms of the data we are able to extract on a comparable, worldwide stage. There are things you can do nationally that we just could not do around the world. We rely on public sources of information. We rely on institutions submitting data to us and also data that we collect ourselves. There are limits to what you can measure in terms of a global stage. Sorry, there was a third question, which I was going to come to.
Q109 Chris White: You have answered all three. You have been lobbied to change the criteria.
John Gill: In terms of year on year, one lesson that should be learned is that consistency is really important when you are measuring things. We all know that, but it is worth repeating, because you need to be able to benchmark year on year and see how performance is changing. We have been publishing the THE World University Rankings for 11 years, and we are in our 12th year now. We developed the methodology with the sector; we went out and spoke to universities, academics and strategic university leaders, and said, “What should we be measuring? How should we be measuring it?” We developed that in a very collaborative way with institutions, which ensured their buy‑in and ensured the credibility of the rankings, and we have stuck with that methodology for 10 years.
It is important to say that, while you need consistency, you also need to make improvements and evolve when there are improvements to be made. Do not just stick with something in a bloody‑minded way because it is the way we have always done it. For example, this year, we switched our data provider: we moved from Thomson Reuters to Elsevier, because we believe that that gave us a larger set of citation information. We were accessing a more global picture of performance, perhaps slightly less US and Anglo‑based research information.
We brought a lot of the data collection that we do in house, so we are now able to be more transparent with the data that have at our fingertips. Also, we expanded our reputation survey and put it out to a large number of professors in different regions around the world to ensure we were getting a very representative global picture. We have been willing to make changes like that, but, in terms of the fundamental criteria that we are looking at, we have really tried to keep consistency as a core of what we do.
Q110 Chris White: From somebody in the editorial department having the brilliant idea to set this up to the first publication of your results, how long was that process? How long did it take to set this up?
John Gill: I was not around 12 years ago. The point I would take from it is that there is a lag in data. This is a point that Nicki would perhaps be better at making than me, but it is important to understand that, when you are looking at metrics and data, you are looking at information, typically, from a couple of years ago. It is not current, so to make key financial decisions for institutions on the basis of data that is a few years out of date, when they may already have done something to address the issues, if there are issues, is another potential problem with what we are doing here.
Nicki Horseman: Given that the gain being measured is from the student’s study and they are studying for three years, if you are measuring employability six months after graduation, what you are measuring is already three or four years out of date and things could have moved on. The TEF’s longevity is three years, I think, so it could be nearly six years in terms of keeping what you are seeing as input to the student as a metric for whatever consequences there might be. That is quite a lag. We have a bit of a lag and we were challenged on that in terms of the rankings data, but that is quite a long time. So much can happen in seven years.
Matt Hiely‑Rayner: We are frequently lobbied to introduce a retention metric and it is something we would like to do. I am concerned to see it in the TEF, because there are several reasons why we do not use it. First, there are multiple outcomes for different students and it is difficult to say, “This is good and that is bad”, when you look at the students that exit with a different award than they set out to get or perhaps transferred to another institution. In some cases, those are good outcomes. Secondly, it is predicted by entry qualification, so you would need to take a benchmark approach to it. Thirdly, this question of lag comes up again. You need to look at two academic years of data in order to assess whether a student got past a census point.
That is only if you are looking at first‑year retention. If you do only look at first‑year retention, you motivate institutions to get students past a census point, even if you are prolonging the agony before an evitable dropout. Retention is a problematic one and, if this is going to be used as the main metric for new providers coming in, you may find that they say all the right things when moving for university status and getting the early awards of TEF. It may be several years before it becomes visible that their retention rates are atrocious and are costing the taxpayer a lot of money.
Professor Wilsdon: On your point about the reputation and things like the Times Higher, John is right that it does rest not entirely but primarily on research. For me, it raises a bigger issue when one takes the totality of the reforms in the Green Paper, which is a danger of pushing things too far in the other direction.
It is quite right to introduce greater emphasis on teaching, but, if we do not think through very carefully the incentive structures that are put in place now on the TEF side, another worry is that, with whatever else happens on the research side—and there are lots of other reforms proposed by the Nurse review and elsewhere—in totality, we will see a teasing apart of many of the interdependencies that exist in the system between research and teaching, which are critical to the health of universities as institutions and indeed to the learning experience. These might be teased apart if you create two more defined, discrete assessment systems on either side, operated by the Office for Students on one side and Research UK on the other. A lot of the connective tissue in the system, which previously was embodied in HEFCE, may disappear or decline, and that needs to be protected. Again, the Green Paper says that, so it is not that this is not recognised, but it is a concern.
Q111 Chair: Matt, in answer to Chris, you mentioned new entrants. That is a big part of the Green Paper. Given what John and Nicki said about lagging data, how can we reconcile this priority towards encouraging new entrants into the sector while, at the same time, making sure that data, common sets of metrics, are as comparable as possible? That is to everybody, rather than just you.
Matt Hiely‑Rayner: That is a tricky one. I believe the sector is moving to in‑year data collections, which may provide an opportunity, but that will be some way off. The intention is for the new providers to be admitted into the sector sooner than that. I do not have any ready answers about how the statistics could highlight that. The statistics tend to look at outcomes. By their nature, for undergraduates, you need three years of provision before you can look at them.
Chair: John and Nicki, do you have any comments on this in terms of your world experience? Is there any insight you can give us?
Nicki Horseman: I agree with Matt; it is difficult to see. The data for the UK is generated by HESA returns, which is a process in itself that closes at the end of the academic year. If there were different metrics being developed, there is a possibility there of taking a different view that might have more currency. For example, if you were talking about qualifications of lecturers and the training they have undertaken, that can be done on an in‑year basis. But anything that is an outcome measure—
Q112 Chair: This is what I am trying to get at: is there a possibility, within the TEF, without it being too bureaucratic, for us to use real‑time information and data as much as possible? Is that at all feasible in a teaching excellence framework?
Nicki Horseman: In terms of what is currently done, it is clearly not covered. It would have to be something new and introduced. I suppose it could be an expansion of the National Student Survey, more in line with the NSSE, as it is known in the US. That is something that could be done. NSS data is relatively current; it is that year’s finalists. That is something that could be overlaid.
Q113 Chair: James, at the start of giving evidence, you mentioned perverse consequences and gaming the system. I understand that we do not know what the unknowns are because we do not really know which metrics will be included in the TEF. Could you give us some advice—not just you, but the whole Panel—in terms of what the perverse consequences could be? How do we minimise gaming in the system as much as possible?
Professor Wilsdon: We have heard, both here and from previous witnesses, some of the examples. Clearly, indicators around employability could introduce perverse incentives. Others on learning gain could lead to grade inflation. As I say, on the research side there are ample examples of indicators that have, progressively, over time, created more problems.
Bear in mind that academics are a clever bunch of people. They will optimise behaviours to achieve the best possible outcomes against the indicators that are being used to measure them. Again, it is not that any one metric in and of itself is inevitably problematic; it is how one uses them together or in combination.
Q114 Chair: As an example, very briefly, if we are going to have, within the TEF, graduate salary levels, does that not have a perverse impact in terms of what we need to do for higher education and widening participation? You are not going to recruit students, perhaps, from disadvantaged background.
Professor Wilsdon: Yes, indeed, as was said by previously witnesses. Of course, if you introduce countervailing indicators around widening participation, then one hopes that, in some sense, the system balances itself out. For me, it goes back to the earlier point I made, which is that you are seeing a lot of quite radical reforms being introduced simultaneously here, across the system as a whole. I completely understand the motivations behind it. Many things in the Green Paper make good sense, but there is a need for more considered testing and piloting of the various indicators that are being proposed so we can understand better the sorts of consequences that do arise. That is something that has been done quite successfully on the research side and we learned from it.
Chair: Do we have any other comments regarding gaming the system?
Matt Hiely‑Rayner: Yes, several. On salaries and other outcomes for graduates, it is important to have a prediction based on the students’ characteristics before they get to university and judge each provider against that. That does not give you a nice, meaningful statistic that is easy to portray to an audience, but it is the fairest way to do it.
On retention, it is important to look at the full course of study, rather than just a one‑year snapshot. When you get down to discipline level, there are also several pitfalls to look out for. The Guardian is unique among the domestic university guides in that it starts at the subject level and builds up to an overall picture of universities. Really, our primary focus is at the subject level. We encounter quite a lot of gaming, with people trying to mess around with the subject codes that are used to identify activity, trying to hide poor provision in places where it will not contribute to their overall ranking. You will see a lot of that when TEF awards go down to discipline level.
Fourthly, on grade inflation, there are some dynamics that can counterbalance it. That is why we have not seen such grade inflation up until this point. Universities are keen to maintain their reputation and they know that, if employers recognise their graduates who have a first or 2:1 as being not very good, it will get to them in the long run. Any system that looks at student achievement through degree classifications or a GPA should also seek to amplify those dynamics.
John Gill: You are right to be worried. Government policy over the last six years has sometimes seemed like the nursery rhyme: there was an old lady who swallowed a fly, and then you have to do something else to counteract the unforeseen consequences and so on. You end up with this slightly ridiculous situation. That is a worry here.
My suggested fix would be to introduce a human element to it to try to soften the edges of the metrics. I know there are plans for the expert panels that will be involved. Certainly in the REF, that has been an important part of the process, having the peer review. Obviously, it makes it more cumbersome and expensive to do as an exercise, but having some human element, where you have practitioners, teachers, academics and students involved in sense‑testing some of the results coming through from the metrics, is really important.
Nicki Horseman: Following on from the discussion about salary, clearly universities’ job is providing people who do not just get good salaries but provide social benefit, and they can be on lower salaries, so that is a difficult measure. Most indicators can be gamed or played in one way or another. Universities are good at that. For example, for learning gain, you start thinking about under‑reporting your students’ qualifications—not the ones that you have admitted them on, but the extras that add up to tariff marks, etc. There is quite a lot that could be done. Some of those things can be headed off by, for example, the Destination of Leavers from Higher Education returns being audited. There be may be a role for more auditing, checking or cross‑checking of data that is submitted.
Q115 Paul Blomfield: I wanted to ask about one aspect of the issue that has come up quite a lot in our discussion so far with previous panels. Matt was alluding to it in terms of the work done by The Guardian. That is subject versus institutional assessment. In contrast with secondary education, where people are choosing a school, a prospective student is focused first on subject and then on institution. In that context, does it help student choice for the TEF to be focused on institution rather than subject?
Matt Hiely‑Rayner: If TEF is going to recognise some of the institutional supports for students, yes, it does make sense. But to look at metrics, even at institutional level, there would have to be subject benchmarking. The data is going to have to go to subject level even when you are looking at an institution; otherwise you will get very different judgments.
Q116 Paul Blomfield: If the data has to go to subject level to get a rounded impression, in the interests of transparency and informing students, is it not the subject‑level data we should be sharing, rather than the institution‑level data?
Matt Hiely‑Rayner: They have that through Unistats already. When you are making a TEF award to an institution, the data that is going to be consumed has to be at a subject level; otherwise you will always judge against conservatoires of arts.
Q117 Paul Blomfield: Is that really helping student choice, if student choice is focused on subject?
Matt Hiely‑Rayner: Yes, because otherwise you will have TEF award 1s for your specialist arts institutions, because they tend to have lower salaries and other outcomes that would not look so positive next to a more technical institution.
Paul Blomfield: What does everyone else think?
Nicki Horseman: It has already been mentioned today that universities are not always strong across the board. They will have some subjects they are better at than others. If that is not recognised, how is it helping students to choose? As Matt says, there is already Unistats and the Key Information Set data that is out there, which is sort of helping. It could be quite misleading to have an institutional‑based overall score. Clearly, when you are talking metrics, subject plays such an important part that it has to be in the mix in terms of benchmarking or any clever, subtle ways of comparing institutions as put together. But there is an issue there that institutions find it hard to report consistently at very fine subject level. It is not easy.
Q118 Chair: There is a final question from me, if I may. James, I was very interested by what you said at the start and throughout, where you questioned whether the timetable could be achieved. You said there was a need for more considered testing. Yes or no would be helpful, but this question is for all of you: do you think the current timetable for the implementation of the TEF is achievable and will provide a meaningful teaching excellence framework?
Professor Wilsdon: It is highly ambitious.
Chair: You sound like Yes Minister there.
Professor Wilsdon: In the way it has been proposed, with a modest first phase, it is achievable. How much it will tell us that is meaningful is open to question, for all the reasons we have been discussing.
Matt Hiely‑Rayner: I think it will be very limited in its early iterations.
John Gill: I agree with James. It is achievable as it has been set out, but whether that is worth doing I am not so sure.
Nicki Horseman: I agree. I think it is achievable, because a lot of the information already exists. What is actually done with it and the consequences could be quite distorting, first time round.
Chair: Thank you very much. That was very helpful.
Examination of Witnesses
Witnesses: Professor Joy Carter, Vice‑Chancellor, University of Winchester and Chair, GuildHE, Professor Dave Phoenix, Vice‑Chancellor, London South Bank University and Chair, million+, John Widdowson, Principal, New College Durham and President, Association of Colleges, Professor David Maguire, Vice‑Chancellor, University of Greenwich (on behalf of University Alliance), and Professor David Eastwood, Vice‑Chancellor, University of Birmingham (on behalf of the Russell Group), gave evidence.
Q119 Chair: Good morning. Thank you for coming to give evidence to us. Could we start by you all saying who you are and which organisation you represent?
Professor Carter: I am Joy Carter. I am Vice‑Chancellor of Winchester University and I am representing GuildHE today.
Professor Phoenix: I am Dave Phoenix, Vice‑Chancellor of London South Bank University, representing million+.
John Widdowson: I am John Widdowson, Principal of New College in Durham and President of the Association of Colleges.
Professor Maguire: I am David Maguire, Vice‑Chancellor of the University of Greenwich. I am here today representing University Alliance.
Professor Eastwood: I am David Eastwood, Vice‑Chancellor of Birmingham. Until September, I was Chair of the Russell Group.
Q120 Chair: Thank you very much for coming. The Minister says that teaching in HE is patchy. Is it?
Professor Carter: There is a lot of good evidence to suggest that teaching in UK higher education is excellent. Some of that comes from the Quality Assurance Agency, some from the NSS scores and also some from what is said internationally about the quality of our higher education. If you look at anything, there will be areas of super‑excellence and areas that need improving. However good anything is, continuous improvement is very good to have.
Professor Phoenix: Every institution in the country is always seeking to enhance what it achieves. If you look at the global level that the TEF is aiming for, across the sector, most institutions are getting student satisfaction levels above 80%; most institutions are getting students in employment or further study levels above 90%. In almost any other sector in any other country, we would be celebrating what we are achieving through the sector rather than trying to disaggregate to a very reductionist level.
John Widdowson: I suppose the problem is that we do not really know. The Minister gets the evidence from the sources Joy and Dave have spoken about, but my understanding of what is being discussed around the TEF is to try to find a way of putting hard data around it. There are some problems with that. My own institution went through the Ofsted process for many years; that is flawed as well. There can be a lot of time and energy spent on this, when institutions themselves may be the best determiners, working with their students and the employers to whom those students then progress.
Professor Maguire: All the evidence suggests that UK higher education is a high‑quality system, based on all available data and by comparison with other areas. It is true that it is not uniformly excellent across all areas and that areas change all the time. Just because something is excellent this year, it does not mean it will necessarily be excellent three years from now. It would be inappropriate for us to try and persuade others that we are universally excellent. We are good, but we can do better.
Professor Eastwood: Whenever we have created a methodology to assess the quality of teaching and learning in higher education, it is suggested that the overall quality is higher. That was true of TQA in the 1990s and the early part of this century. It is true, as colleagues have said, of the NSS. We have data that suggests that the overall quality of teaching and education across the system is high.
I work in other jurisdictions. Certainly, the reputation of UK HE stands very high indeed. None of that, of course, means there is no unevenness, going back, Chair, to your question. The issue, then, is how we best address that. An urgent presenting question would be: “Are institutions indifferent to that variation in the quality of teaching in their institutions?” All the evidence is that, far from being indifferent, they are absolutely committed to consistent improvement in the quality of their teaching.
Q121 Chair: Is the Minister wrong? The Minister is investing time, energy, political capital and possibly legislative time in order to carry this out. It is a key priority of his. Given what you said in your opening remarks, given that teaching excellence is high in this country and that there will be provision that needs to be improved but, ultimately, it is the envy of the world, why is the Minister investing so much capital in this?
Professor Phoenix: The Minister is right to ask the questions he is asking. It is right to challenge the sector. As a whole, the sector is always trying to seek to enhance what it is achieving. For me, personally, the challenge is trying to understand exactly what issue is being addressed. When people talk about “patchiness”, what exactly do they mean? When they talk about “teaching quality”, what exactly do they mean by that? For me, the overall experience of the student is about the experience that university environment gives them. Therefore, it is about how we oversee the totality of that within what is a very diverse sector.
John Widdowson: If it is couched in consumer terms, there are questions about who the consumer is here. My criticism would be that this is about the process of teaching. There is something in there about learning and learning gain, but, as members of the Committee will know, I quite often shift the focus to part‑time students. A lot of debate here is around full‑time students and what is good for them. Part‑time students occupy a different part of the field and have different requirements.
That leads me into then saying, “What is the role of the employer in this?” If the teaching is excellent, but produces people who do not fit what some employers want, clearly there is something to be addressed from that perspective. That is not for every course, but for those who are able to do that. I am not sure that what is outlined at the moment captures that.
Professor Maguire: We have a high‑quality sector, but any sector in any area of the world that rests upon its laurels and does not continuously seek to improve will not be doing justice to all of the stakeholders. We need a system that focuses on the enhancement and improvement of learning and teaching.
There are five key principles we should put at the heart of that. The first one is a focus on enhancement and getting better. The second one is about reputation. We need to ensure we seek to enhance and certainly do not weaken the reputation of our world‑class sector. We need to preserve diversity. There is not one model of teaching excellence or of success. We need to recognise the different missions and different goals of different areas. Our sector is founded upon autonomy and the autonomous nature of institutions. We need to preserve that. We also want something that will be efficient. We do not want to spend a lot of money and time and deflect people from doing research and teaching by concentrating on filling in forms and delivering information.
Chair: I must say, David, those principles you have outlined are incredibly helpful to us as a framework by which we can evaluate what the Minister is trying to do. I would like to thank you for that.
Professor Carter: I will echo my colleagues and say that continuous improvement is already at the heart of UK higher education. What is proposed may help us to increase the rate of it, but must not forget we are all engaged in it anyway. What is proposed may also raise the status of teaching, which is important too. But, clearly, what we are asked to do must not be too burdensome, and that is really important.
Q122 Chair: Can I push you on raising the status of teaching, Joy, before I bring Paul in? Yesterday, as a Select Committee, we had a visit to Birmingham. We were in Birmingham City University. There were other universities there, and somebody said—not from Birmingham City University, although there was a common consensus about this—that you could be a Nobel prize winner in your respective field, but, unless you were embarking upon a teaching qualification and you were emphasising teaching, you would not get promoted in that institution. That really struck me. It shows how important teaching is in the HE sector and how it is coming up the agenda. Is that a fair summary of what happens across the board or is that an isolated case?
Professor Carter: All universities throughout the sector are becoming more aware of the importance of teaching and are recognising that in progression, in awards, in lots of different ways. There is already a movement to do this.
Chair: Is that common across the board?
Professor Eastwood: Every Russell Group university I know of, in its promotion criteria, will promote people on a teaching‑led track. All new entrants to the profession over, broadly speaking, the last decade, if they did not have a teaching qualification, would go through an accredited programme in their own institution. There is no doubt that there has been a very strong emphasis on equipping faculty as teachers over the last decade. There are also myriad processes in most institutions to recognise teaching excellence, both as defined through peer review and, certainly in my institution, as recognised by students.
Professor Maguire: Speaking on behalf of the alliance, which represents roughly 25% of all of the students in this country, I would concur with David’s observations and say that all staff are really serious about the importance of teaching. There are promotion, reward and recognition schemes in all of our members. At my own institution, two thirds of my staff have a formal teaching qualification and our goal is to get to three quarters within 18 months.
Professor Phoenix: Across the sector, you will find a focus on a level of expectation with respect to teaching and support. There is as well, though, the opportunity within the UK for people who really want to focus on teaching to get those external measures of excellence that support promotion. For example, we have the National Teaching Fellowship framework. We have a range of activities through the HEA, Higher Education Academy, and other bodies where people who want to be sector‑leading in terms of the development of new teaching methodologies can get the same external recognition that can feed into promotion rounds. There is a standard support within the sector and there are also activities outside of institutions to help people build profiles that contribute to cases for further promotion.
Q123 Paul Blomfield: I wonder if I can ask three interlinked questions to each of you. The starting point is that we have an incredibly strong university sector. Part of that strength is the diversity that each of you represents. In that context, are a common set of metrics going to provide the sort of information the Government wants and students want? What do you think of the three metrics that are proposed so far? There seems to be a shifting the debate between quantitative and qualitative. What is the balance between those metrics and narrative reporting? What lessons can we learn from the evolving REF in that?
Professor Maguire: Going back to those principles, one of the challenges is optimising all of them simultaneously, preserving the diversity of the sector. The challenge we have is that, if we use a small number of simple metrics, we do not unpack the sophistication, the complexity and the diversity of the sector. On the other hand, if we go for a sophisticated array of very detailed metrics, supported by peer view and self‑evaluation, we end up with a very big, complicated and costly sector that transgresses one of the other principles identified, which is to minimise bureaucracy and cost to the sector.
It is a very difficult problem to solve. We can go for something simple and high‑level, which is subject to perverse use and gaming; or we can go for a much more sophisticated and detailed thing, which may still have some problems but will be very expensive, costly and time‑consuming for the sector.
Professor Eastwood: This relates to a point David Maguire made earlier. The key question we should ask ourselves is: “How good are universities at what they seek to do?” That recognises the diversity of the sector; it recognises the diversity of the student body; and it recognises the autonomy of universities in the way in which they shape their academic and vocational programmes and a distinctive student experience. Any TEF that is mature will have to have ways of recognising that difference.
It is really important that we recognise difference and we do not turn difference into a hierarchy. In the UK, it is often said that we have a genius for turning difference into a hierarchy, and rarely is there a week that goes by without a new league table being produced. We all have deep anxieties that overly simple metrics will be translated into league tables, which will mislead students, will have perverse impacts on the way universities and other HE providers operate and will not address the core issue we have all been focusing on: enhancement.
John Widdowson: I will pick up on the point David made at the end there. It is tempting to use university as a proxy for all higher education, but clearly it is not and will become less so in the future. A set of metrics that might work perfectly well within the current university system probably will not work with some of the new entrants we expect to come into the HE world and also some of the activity that is there now, including in some universities; it is not particularly an FE college point. It is around professional and technical education and the different requirements made of teachers in that field, where being active practitioners and bringing people from industry into the classroom, the workshop and laboratory is one of the key challenges. We have to have a metric that covers that. It is almost like a spectrum, with pure research at one end and then very applied things at the other end.
Talking to students in FE institutions studying HE qualifications, they want the teacher to be able to teach well, but they are not very specific about what that means. I accept there are different ways of defining that in different contexts, but they want to be well taught. That is particularly so for part‑time students, who are giving up time from work and are probably being sponsored by their employers. They do not want their time to be wasted, as they would see it, by inefficient or ineffective teaching.
The second thing is that they want those teaching them to be credible practitioners in whatever field it is, having had some real experience in an area that is current. For institutions like my own, for the college sector and for a lot of universities, it is about how you maintain that currency with the world outside. The teaching is in the middle in some ways, and you have to reflect both aspects, both ends of the spectrum.
Professor Carter: At the heart of GuildHE is a belief in excellence in diversity, so I really welcome the question and thank you for it. Is a common set of metrics going to give the right information? No, not on its own. That leads me to a key point, which is that the contextual statement is all‑important, and we would like to see it weighted equally with the metrics. It must not be notes to the metrics, but something real and powerful in its own right.
Professor Phoenix: In responding, could I take a step back from the position some colleagues started from? To answer your question and tell you whether or not I believe those metrics are the best way forward, I need to understand the purpose that you are seeking to achieve. At the moment, while there is a lot of discussion around the teaching excellence framework, I do not yet have clarity on what is meant by that. What is the main outcome?
For example, if it is around the overall experience students have, which ties into social mobility, value for money and the other things the Minister has spoken about, we need to look at the overall learning environment. One of the challenges is, if you take those three metrics, you have gone for a very reductionist approach in terms of three simple measures.
In fact, if you follow through with the Green Paper, one of the concerns I have is that, at the moment, the overall academic environment for a university is overseen by HEFCE. We have reports on all aspects of what we do going into one body, which can balance things. In the new world, that will no longer exist. We will end up with teaching and research. I am unclear where civic responsibility and knowledge transfer will go, along with other aspects that contribute to the experience the student has. We need to know what the question is to answer what the best way to progress is.
Whatever it is, if it is related to that environment, the contextual statement will be incredibly important, not just to recognise the diversity but to give individual institutions with different environments the opportunity to explain how they are delivering their mission and adding value to students. The reason that is essential is because, although there is a lot of focus and discussion about whether we should have learning gain, diversity itself creates problems even with that measure. We have to look at local institutions and whether they are delivering what they are promising to students.
As a very final, quick point, what is the value add? All those measures you have talked about are already available to students. If you go to any of our webpages and look at a course, you will find a link to a Unistats site that tells you how many hours of contact there is, what the graduate starting salaries are, what the employment levels are, what the entry qualifications are. That is all available to students now, through the investment universities in the sector have made.
Professor Carter: I wanted to say something about the three measures specifically. First of all, on retention, one might say that that measures something completely different from teaching excellence. Actually, it might be to do with accommodation; it might be to do with how much mental‑health support there is in an institution. There are lots of other factors like that, to do with professional services rather than teaching excellence. On the employment measure, I would really question the link there with teaching excellence. Particularly six months after graduation, there is probably no link at all with teaching excellence, so I urge real caution there.
Student satisfaction is a lot more closely linked with teaching excellence, depending on which measures you take and which questions you ask. That probably has to be weighted compared to the others. I do not know the extent to which these metrics will be weighted, but I would definitely suggest they should be.
Q124 Paul Blomfield: There was a general nodding of heads when Joy said and David echoed the concern about the contextual statement needing to be given equal weight. If I could push a bit more on the quantitative metrics, David, you said that overly simple metrics would potentially mislead students. Are you worried that the metrics that are currently being proposed are overly simple?
Professor Eastwood: I would answer your question in two ways. One is about whether this will mislead students. In my experience, students are very sophisticated; they are sophisticated in the judgments they make. I was running HEFCE when we established Unistats, and I said it would be used unevenly by students. It has been unevenly used by students, because students are making an integrated assessment of the programme, the nature of the programme, the focus of the programme, the institution, other things the institution provides and the kind of higher education experience they want to have. Whatever we come up with, it will be filtered by students in ways that might surprise some people and some Ministers, but will actually be quite intelligent.
As far as the proposed metrics are concerned, Joy is right. They measure a range of things, including the support for teaching and the support for education, as well as the delivery of education, though I would see that as part of an institution’s commitment to a high‑quality student experience. The issue we have to be careful about, which we have not yet focused on, is that, if we put a lot of weight on a basket of exit data, graduation, degree classification, first destination and salaries, those are all legitimate and students have a right to have information about all those things, but it is critically important that we do not incentivise those and disincentive access, widening participation, universities taking risks in whom they admit.
We also must not set up perverse incentives that encourage universities to boost provision in certain areas because they have better outcome metrics and better salaries, and disinvest from areas that are critically important to the economy, society and social wellbeing. That is why I worry. These are all measuring things that, to some extent, are worth measuring. It is about how they are concatenated and then re‑presented as some sort of overall assessment of the quality of a programme or an institution. That is where it becomes dangerous, because that is where it becomes reductionist.
John Widdowson: I have a quick comment on salary as a proxy for anything, really. I cannot see, as Joy said, the direct connection between the excellence of teaching and the salary a student might get in the first or even subsequent destinations. That is about the choice of career pathway. For some students, as well, there may be a perverse disincentive to look at things like working for small companies, because I know they will not pay as much as large companies in general, and setting up their own companies, because earnings are not going to be great in most cases for the first few years. We have to be careful around using salary as the only indicator.
David alluded to this, but there is that general social gain of having people with higher qualifications in local communities, who might be doing things like classroom assistants in a local primary school, but they will be the first people with a high‑level qualification in their community and they will have a responsible job that others will recognise and reflect on. Salary is a very dangerous proxy to take and needs to be moderated in some way.
Q125 Chris White: I was going to ask David to expand his point about misleading students, but your response to Paul’s questions has clarified that. John, the FE sector is very different, as you have already mentioned, whether it is part‑time study, people doing evening courses, people already in employment. How do you see that fitting? Is it possible to fit? Do you have any suggestions of how those different criteria can be included?
John Widdowson: I think there is an overlap; I do not think there is a perfect fit. I always return to part‑time students, because they are the hardest to encompass with any system. As colleagues have said this morning, part‑time students will be less concerned about things like student accommodation and the quality thereof, or about student recreational facilities, because they will not use them. They will live at home and they will have a job, which they will go to more than they will go to the institution. We have to find some way of capturing that.
Linked to that is some measure, at some point, of employer satisfaction with what is happening there. That is more concerned with the output, the outcome and the distance travelled by the individual. It is linked to learning gain, but it is a bit more than that. It is about what makes them a better employee at the end of the provision.
Chris White: Is that about co‑design of courses? .
John Widdowson: Yes, and how you engage with the employer to give another measure of quality. My institution offers a lot of sub‑degree provision as well—higher nationals, foundation degrees—which are intended to be employer‑facing. If they do not have credibility with those employers, we will lose that traction and that market, and the students will not come because they do not see the added value to themselves.
Q126 Chair: May I ask a series of very quick questions? Following on from what Paul was saying and Joy’s and David’s responses as well, should we decouple things like widening participation and the ability to raise tuition fees from the TEF?
Professor Carter: We should think very, very carefully about that. It is an absolutely fundamental question. We need a sustainable sector in financial terms as well as in other ways; that is a given. But there are dangers. There are some huge dangers with widening participation. We have achieved so much for so long. Of course, we have the Prime Minister’s targets to achieve, which are very ambitious, so we must not do anything that gets in the way of those.
Professor Phoenix: When looking at aspects such as widening participation, the challenge is that institutions, in terms of that environment I spoke about, are geared up to provide support for different categories and groups of students. Therefore, if you are looking at environment, aspects linked to social mobility and how it is provided could be key.
I am absolutely clear that the fees should be decoupled. I do not see any basis for coupling the fees and, as yet, no clear advantage has been put to me. If anything, if we can simplify what we are trying to achieve with the TEF, it is more likely to deliver some real benefits to the sector and to the students. If, at the end of this, part of the desire is to open up the market based on quality and price, why have these artificial interventions? Why do we not have a debate about what we need to do to build on this?
John Widdowson: Fees are slightly less relevant to the college sector, in that we already charge, on average, lower fees that most HEIs do. Therefore, an incentive to increase the fees is not going to make us teach any better, or any worse, indeed.
Students have a right to expect excellence. They are paying for it themselves, or their employer is paying for it, and therefore, whatever the fee level, they deserve excellence. I agree that needs to be contextualised; it needs to give them what they expect and what they are promised. But connecting that to any measure of teaching excellence probably only encourages the sort of gaming that has been talked about, for a very small return, actually. We have to focus on what we want to achieve through a teaching excellence framework, rather than any other incentive that might be tagged on for institutions.
Professor Maguire: I welcome the Green Paper’s comments on the importance of widening participation and the Prime Minister’s goal. It is absolutely fundamental to the design, development and implementation of any teaching excellence framework. One of the assumptions behind the TEF, however, is the model of what a student is.
I am afraid much of it smacks of the notion of an 18‑year old with A‑levels who has gone to school and college. That is no longer the reality. Many are older; many are on flexible programmes. We now have the whole development of distance and online learning. How does that translate into the metrics? A whole swathe of the sector has been omitted by the use of traditional metrics. As an absolute minimum, I would echo the requirement for contextualisation and the incorporation of widening participation in that, but anything we can make explicit would also be welcomed by the University Alliance.
Professor Eastwood: I have two points. The first is that the sector has made huge strides in widening participation over the last 15 years. It has done that as the funding regimes have changed. That speaks to the commitment right across the sector to meeting the widening participation challenge.
On the link between TEF and the fee cap, there is an obvious point, which is that, if the raising of the fee cap is done through CPI and you end up with a four‑tiered TEF, these are quite precise judgments that are being made, shall we say? My own view is that decoupling TEF from the fee cap would be sensible. We do have to have a serious debate around funding over the next five years, otherwise, year on year, funding will reduce and a consequence of that will be an erosion of the quality of learning and teaching. I welcome the fact that the Green Paper has put that on the agenda.
My own view is that the best way of balancing the price of higher education and student choice is informed student choice. In so far as this is an exercise in better informing students, that then prepares us for an environment where the fee cap can be moved.
Q127 Chair: David, I will stay with you. We have heard several times this morning and over the course of the inquiry that we have a world‑class sector, that it is the envy of the globe and that we are looked at. In terms of international students and our standing, what impact could TEF have on that?
Professor Eastwood: We have a capacity to talk ourselves down and that is unhelpful. It is partly because we have more advanced techniques for evaluating quality and student satisfaction. The NSS is not replicated globally and most systems would not have the courage, actually, to embrace an NSS.
Q128 Chair: Are we top of the league, then, in terms of making sure there is quality assessment and evaluation centred on the student?
Professor Eastwood: We would certainly be in the top decile. If you look at the way in which quality assurance works across Europe, it would put us in the top decile. If you look at the way in which quality assurance is being developed in Asia, the British model is often used as the baseline. I am an international member of the Hong Kong University Grants Committee, which constantly looks to the UK as its inspiration for improvement. In terms of the frameworks we have created, yes, we are class‑leading or close to class‑leading. That is not an argument for being complacent.
One of the worries quite a lot of us have about our relative global performance as a system is that that position might be quite fragile. I chair Universitas 21, which is a group of 25 global universities. Each year, the University of Melbourne provides us a ranking of higher education systems—not of institutions, but of systems. In the four years Melbourne has done that, the UK has figured between 21st and 26th in terms of inputs—that is what we invest in our system—and it has always been 2nd in terms of outputs. That speaks to a system that is efficient; it speaks to a system that is investing the resource it has well. For some us leading within the system, it also speaks to a fragility. This is why we are anxious about some of the rhetoric that surrounds the TEF. We are anxious that the near‑primacy the UK system has might be under challenge.
Q129 Chair: I am conscious of time, so I am going to ask a final question based on process and bureaucracy. Is it possible to have a light‑touch, risk‑based approach on quality assessments while, at the same time, being thorough, credible and distinctive?
Professor Maguire: In a word, no. I can elaborate on that with the points I made earlier on.
Q130 Chair: Is TEF going to be a bureaucratic tick‑box exercise that drags you down and wastes resources?
Professor Maguire: As I said, there is a spectrum of alternatives here. One is a very high‑level, light‑touch tick‑box exercise, which risks the challenge of gaming but would certainly minimise the cost to the sector—a sector that, as we have said, all the evidence says is performing well. If we go to the other extreme and have something very detailed and sophisticated with a lot of metrics, which has really significant outcomes for the sector, we run the risk of it being incredibly expensive.
Remember that the REF, according to a Technopolis evaluation, cost £246 million to implement. A TEF done on anything like a similar sort of level will have double the submissions and the capacity to cost twice the amount of money. That would not be serving the sector well, when that amount of money can be put into teaching students and doing world‑class work in enterprise and research domains.
John Widdowson: There has to be some recognition of what institutions themselves do. We have heard that from everybody sat at this end of the room today. A TEF that tries to measure too many outputs seems to be wrong, to me. It is going to be expensive and bureaucratic; it will be gamed at some point.
Something that looks at what institutions do, which gives you that ability to look at the mission and specificity of what is happening, is the way to do it. You measure what the institutions do internally, whether it is training of staff in pedagogic methodologies, as they call would it; we would call it teaching in FE. That is the sort of thing that would be useful. I am sorry, David.
Chair: We know how to talk in the north‑east, don’t we?
John Widdowson: We do actually, yes. But that is one of the key strengths of the system. Therefore, any system we have has to go with those strengths. Put that responsibility on to institutions, in the way the quality assurance system is heading, where institutional governing bodies take a view on this and regard it as a key function for them.
Professor Phoenix: Building on that last point from John, we should be clear. I have not heard anybody involved in the HE sector who has had any disagreement with the principles the Minister is trying to progress. Never have I heard anybody in the sector oppose enhancement. Often, actually, one of our challenges is that we are always aiming for the next level, to try to improve.
The two points I would like to make, though, are these. First, if you are going to maintain the autonomy of the institution, there has to be consideration of how, through the accountability of vice‑chancellors and principals to their boards, that allows Government to work through an independent body to make sure they are driving the quality agenda on a national level, but with autonomous and local focus. There is also the concern, as I said before, about what body will look at the totality of that now in the Green Paper.
The second point is this. If you move away from that principle, then, to enable institution‑to‑institution comparison, it is inevitable that you will have to bring in a degree of a national framework. The worry there is that you could end up stifling innovation and stifling diversity, which is exactly the strength that this country has and the powerhouse that has the ability to help it move forward in the future.
Professor Carter: I hope it will work, but there are huge dangers. We have talked about some of them, such as our international reputation and perverse outcomes. The final point I want to make is about staff workload. Something we are all overlooking here is that excellent student outcomes come from satisfied staff with a high wellbeing. I would ask this question: are we focusing too much on the students, without realising that there is a direct link there?
Chair: That is very helpful. We could have gone on for a considerably longer time, but thanks for your evidence. We really appreciate it.
Oral evidence: Assessing Quality in Higher Education, HC 572-ii 2