Written evidence from the Directors of Academic Audit Associates Ltd (QHE 59)

Academic Audit Associates (AAA) Ltd. was recently established to provide academic audit and advice services to higher education governing bodies and senior managers. The company has ten associate members, each of which has substantial experience of university management at a senior level. 

The authors of this submission (Dr David Cairns and Professor Colin Raban) are the founding directors of AAA. 

Dr Cairns joined QAA in 1997, having previously worked with the Council for National Academic Awards and the Higher Education Quality Council. He coordinated institutional and overseas audits. From 2005 he designed and led QAA’s Outcomes from Institutional Audit project and worked on what was then the ‘Academic Infrastructure’. Dr Cairns left QAA in 2009 and he now provides consultancy services for universities and colleges, private sector higher education institutions, providers of professional services, and quality and qualifications agencies in the UK and worldwide

Professor Raban is an Emeritus Professor of the University of Derby, having previously held management responsibilities for quality assurance and academic development at several universities.  His most recent appointment was as Deputy Vice Chancellor at the University of Greenwich. Professor Raban now works as an independent consultant to higher education institutions within the UK and overseas. He has served as a QAA auditor since 1997 and he was the Director of the HEFCE-funded Good Management Practice project on Quality Risk Management.

Executive summary

The submission argues that

• the current QAA review method, adopted at the behest of HEFCE, fails to provide a proportionate and reliable means of securing the accountability of universities and other higher education providers (HEPs)
• the higher education sector now encompasses a variety of HEPs and that more higher education provision is delivered outside universities than within them and that external reviews need to accommodate this variety
• it is therefore necessary to differentiate between providers and review them according to their context and the strength of their governance arrangements
• giving greater prominence to HEPs’ internal governance arrangements would encourage them to pay greater heed to their existing responsibilities for safeguarding academic standards and enhancing the quality of their provision
• in line with developments in higher education since the millennium, the Teaching Enhancement Framework needs to be recast as a ‘Learning and Teaching Enhancement Framework’ (LTEF)
• the objectives of an LTEF should be to identify excellence in learning and how teaching and learner support can best enable excellent learning and inspire HEPs to  emulate and surpass each other
• HEPs’ claims for learning and teaching excellence should be independently verified by expert peer panels whose findings are published and periodically checked.
• the link with fee levels should be abandoned in order to avoid damage to the global reputation of UK higher education.

What issues with quality assessment in Higher Education was HEFCE’s Quality Assurance review seeking to address?

1             This question is best answered by HEFCE and by the other funding bodies (DELNI and HEFCW). As we understand it, three issues form the background to the current review:

i                  The commitment of the present Government and its immediate predecessors to the development of a competitive market for higher education with regulatory arrangements that constitute a ‘level playing field’ for public and private providers. [1]

ii                Concern with the consistency of academic standards both within and between institutions[2] and, more recently, concerns about the quality and probity of some institutions, particularly those trading at what is perceived to be the lower end of the higher education market.

iii              The need to reduce the (actual or perceived) ‘regulatory burden’ on higher education providers (providers or HEPs).[3] Prompted in part by the 2011 White Paper, the Quality Assurance Agency (QAA) has sought to develop more ‘proportionate’ and ‘risk-based’ approach to external quality assurance.

2             From 2011, QAA adopted new review methods that tested a provider’s compliance with externally set ‘reference points’ (restated as ‘requirements’) rather than auditing institutions against their own professed standards and procedures, and their capacity to apply them.[4] The design of the new review methods were strongly influenced by the English Funding Council, the report of the 2009 Select Committee and demands for a means for the identification of ‘bogus’ providers.

3             The result was of this change was[5]:

• A single method of external review (HER), with minor variations for different types of provider and in accordance with perceived levels of ‘risk’.
• An attempt on the part of QAA to act as a regulator rather than the instrument through which the higher education sector regulates itself: the Agency has thereby redefined the purposes of external review and it has altered the division of responsibility between itself and institutions.[6]
• HER exercises are undertaken by teams drawn from a cohort of reviewers that is larger and more diverse than had hitherto been the case.[7]
• To achieve some measure of consistency of practice on the part of what is now a heterogeneous cohort of reviewers, teams are required to use the UK Quality Code for Higher Education as a compliance checklist.
• The new Quality Code has developed since 2009 to serve the needs of what is now a diverse collection of higher education providers (HEPs). The Code provides the criteria against which alternative providers and other institutions are assessed for the purpose of conferring or confirming their status as Highly Trusted Sponsors.

4             We think that the single review method, together with the use of the Quality Code as a checklist, is largely responsible for the antipathy to HER that is expressed by influential universities. The review method itself is perceived to be formulaic and prescriptive. We have worked as consultants with a number of universities and alternative providers and, in our view, HER is not cost-effective: it imposes a heavy burden on institutions whilst failing to generate significant benefits (in the form of useful and credible reports) for the institutions under review. HER is also a method that is all-too-amenable to ‘window dressing’ and ‘game playing’ and, as such, it may not be capable of providing the reliable assurances that are required by the funding bodies and other external stakeholders.

5             In short, we believe that the current arrangements do not make a significant contribution to the lessening of the ‘regulatory burden’, nor have they strengthened the accountability of institutions.

Will the proposed changes to the quality assurance process in universities, as outlined by HEFCE in its consultation, improve quality in Higher Education?  

6             Two points need to be clarified before we answer this question: the Quality Assessment Review concerns the full range of HEPs and not just the universities; and the funding bodies are proposing changes to the external quality assessment of universities (and other HEPs) rather than making recommendations for change to their internal quality assurance processes.[8]

7             If they were to be implemented, we believe that the funding bodies’ proposals would have a positive direct impact on the external quality assessment of institutions and a beneficial indirect impact on the design and operation of their internal quality assurance systems. The proposals may also lead, indirectly, to improvements in the quality of provision, and perhaps also to the standard of awards.

8             HEFCE’s June 2015 publication (Future approaches to quality assessment…) is a complex and confusing document, and some of its proposals lack clarity. In essence, however, the funding bodies are proposing to place a ‘greater reliance on an institution’s own review and governance processes’ and to reduce significantly the ‘requirement for the cyclical review of baseline compliance conducted by an external agency’. Let us deal with the second point first.

9             The planned reduction in the ‘requirement for … cyclical review’ is consistent with the government’s long-standing commitment to lightening ‘the regulatory burden’. In this respect, HEFCE’s proposals represent a break from its own earlier (and QAA’s continuing) interpretation of the principle of ‘proportionality’. This was based on the premise that regulation and external review should be governed by (that is, they should be proportionate to) an institution’s ‘track record’: minor adjustments can be made to the frequency and intensity of the single (‘one size fits all’) review method depending on some post hoc judgement of ‘risk’. HEFCE’s June document, on the other hand, proposes a different approach, one in which the funding bodies’ engagement with an institution, will be appropriate (proportionate) to its type or character.

10         We have argued elsewhere that a single method for reviewing all HEPs is inequitable and inefficient, mainly because it fails to recognise the differences between institutions in a diverse higher education sector.[9]  By contrast, HEFCE’s proposal to place a ‘greater reliance on an institution’s own review and governance processes’ will lead, necessarily, to the development of review methods that recognise considerable variations, from one provider to another, in the maturity and capability of governance and internal review systems.

11         The June consultation document recognises the distinctiveness (and perhaps the need for the differential treatment) of ‘autonomous’ and ‘established’ providers. The former possess their own degree awarding powers (DAPs) and their governance arrangements have been tested through the DAP scrutiny process and/or by successive QAA audits. And the latter, will have met the governance requirements for designation as institutions that are eligible for HEFCE funding. Because the sector comprises institutions of very different type that we would question the appropriateness of the ‘level playing field’ metaphor .[10]

12         Our experience leads us to put a high value on safeguarding the autonomy with respect to academic matters of chartered higher education providers and those that have achieved DAPs since 1992. Correspondingly, we are opposed to any attempt to undermine that autonomy and dictate to academic staff how they should research, teach and support learning, whether that takes the form of a prescriptive checklist badged as a ‘Quality Code’ or an ex cathedra insistence that teaching has primacy over learning.

13         It is for these reasons that we think that HEFCE’s proposal to build future quality assessment arrangements around HEPs’ own academic and corporate governance arrangements deserves careful consideration. HEFCE has endorsed the Higher Education Code of Governance produced by the Committee of University Chairs and we agree with HEFCE that this Code represents a proper statement of the responsibilities of governing bodies in institutions that have the power to make higher education awards. We also agree that according a greater role to HEPs’ own internal governance and oversight arrangements for their academic and other operations puts the locus for quality assessment and academic standards where it should be: with those who are formally accountable for the activities of the HEP they govern and serve.

14         When signing the Intermediate Range Nuclear Forces Treaty, President Reagan quipped: ‘doveryay no proveryay’ – ‘trust but verify’. The phrase applies equally well to HEFCE’s new approach which it defines as ‘the operation of assurance processes by autonomous providers and by the sector as a whole … and the testing of assurances about these … activities by the relevant funding body’.

15         HEFCE’s June proposals imply a degree of trust in the capacity of an HEP with degree awarding powers – such as a university – to manage its responsibilities for the standards of its awards and the quality of its provision. This is welcome and, as we have stated elsewhere, we hope that it will form the core of any new policy mix. HEPs with awarding powers will, however, need to honour their side of the bargain. The onus will be on them to demonstrate that they are applying their creative energies to the development and evaluation of internal quality management and governance arrangements that command the active and willing support of their staff and which are cost-effective and fit for their own purposes.

16         This will be no mean task. For too long, the higher education sector has been beset by a compliance and audit culture. HEPs have responded to external requirements by adopting bureaucratic approaches to quality management that are based on a lack of trust, and which undermine the professional commitment and motivation of staff. Internal quality assurance systems are all-too-often regarded as ‘beasts to be fed’: staff respond by adopting ‘satisficing’ behaviours and by game-playing; resources and effort are diverted from the core activities of teaching and research, and thus the effort to assure quality impedes its achievement and enhancement.

17         We believe, therefore, that HEFCE’s proposals will make a positive indirect contribution to the improvement of quality by encouraging HEPs to develop quality assurance systems that are fit for purpose and not merely compliance-driven. Robust internal processes and governance arrangements are vital if we are to have confidence in the ability of providers to secure academic standards and enhance the quality of their provision into the longer term. We acknowledge, however, the complementary role of the proposed Teaching Excellence Framework in affecting more directly programme quality.

What should be the objectives of a (Learning and) Teaching Excellence Framework (‘TEF’)? 

18         We understand the Minister’s wish to rebalance the attention that universities give to research and ‘teaching’ in favour of the latter. We are concerned, however, that the focus in his statement was entirely on teaching rather than on ‘learning’, teaching and learning support.

19         The UK model of higher education is predominantly learner-centred: that is, learners undertake much of their learning independently, but with guidance by academic tutors and learning support services – such as libraries, IT services and other learning support workers. Supported by these staff students assume greater responsibility for their own learning as they progress through their studies. We think that a Teaching Excellence Framework that reasserted the primacy of teaching over learning would be a retrograde step. It would drag UK higher education back towards teacher-dictated and teacher-led models of higher education and it would fatally diminish the high reputation UK higher education enjoys globally and reduce its attractiveness to high-achieving UK and overseas students.

20         We therefore think the title of the Teaching Excellence Framework needs to be be re-stated as a “Learning and Teaching Excellence Framework” (LTEF). Its objectives should be to identify what constitutes excellence in learning and how teaching and learner support can best enable excellent learning.

a. How should a (L)TEF benefit students? Academics? Universities? 

21         Benefits to students. A (L)TEF should enable students to identify whether the support and tuition they are receiving is excellent and is enabling them to undertake excellent learning. It would provide a new focus for students to engage with their higher education provider to enhance their learning, and (as applicants) to choose a HEP that will enable them to achieve their potential.

22         Benefits to academics and other staff involved in supporting learning. In UK higher education today support for learners in most HEPs is provided by teams of academically qualified tutors and practitioners, ideally operating at the forefront of their subject, working with professional learning support staff .

23         In our experience tutors and professional support staff are keen to support students to achieve their potential and open to advice, guidance and support to enable them to enhance their own performance and that of their students. A (L)TEF would be welcomed by them as a source of information and inspiration, to assist them in supporting students as effectively as possible with the resources available. It would also support tutors and professional support staff in their personal and career development.

24         Universities and other higher education providers. The greater part of higher education in the UK is now provided outside universities; hence we have taken the term ‘universities’ as shorthand for ‘all higher education providers’.

25         In our experience the better HEPs (including those without degree awarding powers) strive to enable their students to achieve their potential and actively seek for ways to enhance how they can best support their students to learn. A (L)TEF would provide the leaders and managers of such HEPs with valuable information on the best of current practice and spur them to adopt it or surpass it. Published information on excellence in learning and teaching derived from a (L)TEF would enable providers to differentiate themselves from those less proficient and attract increased applications from students –which serve as an added inducement.

b. What are the institutional behaviours a (L)TEF should drive? How can a system be designed to avoid unintended consequences?

26         A well designed (L)TEF should encourage providers to foreground their support for learning and for enhancing the effectiveness of their staff to the same degree that they promote their achievements in research. In turn, that should lead to learning and learner support (understood broadly) being granted greater status and salience in the HEP’s decision-making.

27         Unintended consequences. If it is introduced as a “Teaching Excellence Framework”, which we would deprecate, this would be the third attempt in England Wales and Northern Ireland that has been made to measure and report on “teaching”.

28         The previous two attempts were Teaching Quality Assessment (TQA) which was developed by HEFCE. The second, Subject Review, was developed from TQA by HEFCE and then handed over to the QAA to operate, They were of their time and deferred to the primacy of teacher-led learning.

29         TQA and Subject Review did have some beneficial effects but they also prompted institutions to expand the resources they devoted to ‘compliance’ activities which, as an observer noted at the time, created work but not quality or improvement in the higher education that they offered. Both TQA and Subject Review were relatively easy for HEPs to manipulate, their outcomes were distorted by the reluctance of reviewers to criticise august institutions, even when criticism was merited, and they were costly to operate for HEFCE and QAA, and costly also for the largely sceptical providers that were visited by TQA and Subject Review teams.

Avoiding or reducing unintended consequences

30         The higher education sector in 2015, especially in England, is more heterogeneous than at any previous time and there has been a sharp rise in the number of students learning with ‘alternative providers’. Since 2008 and particularly since the raising of fees under the Coalition Government, the influence learners expect to have with HEPs has increased and, while some balk at the notion of learners being ‘consumers’, they have plainly become more assertive of their ‘consumer’ rights when the quality of learner support and other services is being discussed, whether with an individual provider or across the higher education sector.

31         For any new (L)TEF we propose that evaluations of providers’ claims for learning and teaching excellence should be undertaken by expert peer panels of academics and learning support professionals, the members for which should be identified in much the same way that members were identified for the panels that operate under the Research Excellence Framework and might include leading international experts. In the case of the (L)TEF we think there is also a case for learners’ representatives to participate in the work of the panels as observers and guarantors of their integrity.

32         We believe that the use of expert panels should ensure that academic and professional judgements about excellence are made by persons who will be acknowledged by their peers to have the expertise to make such judgements. We think that claims for excellence in learning and teaching should be made by means of written submissions with portfolios of supporting evidence.

33         If our advice is taken and a ‘Learning and Teaching Excellence Framework’ is implemented there may still be some unintended consequences. We think that these could be reduced if the eventual proposals are subject to meaningful consultations with HEPs, experts and students. We also advise piloting the apparatus for the (L)TEF across a representative sample of providers, to test the panel methodology, for example, and to allow for analysis and improvement before full deployment.

c. How should the effectiveness of the (L)TEF be judged?

34         We think the (L)TEF should be evaluated for value for money by the National Audit Office at an appropriate point and, after three years, for its effectiveness in promoting its agreed objectives. The effectiveness review should be undertaken by a panel of independent experts, including international experts, in much the same way that the Research Excellence Framework has been evaluated.

How should the proposed (Learning and) Teaching Excellence Framework and new quality assurance regime fit together? 

35         We have proposed (above) that the main task of evaluating claims for excellence in learning and teaching should be undertaken by expert panels. We think that whatever form a future Quality Assessment Framework takes it will need to provide for some means of cyclical external review of the effectiveness of academic and corporate governance of HEPs. These checks should include consideration of the effectiveness of the approach each HEP has taken to achieve excellence in learning and teaching and form part of the published reports of the cyclical reviews.

What do you think will be the main challenges in implementing a (Learning and) Teaching Excellence Framework? 

36         We envisage two main challenges. The first is to produce a framework for the exercise that is properly aligned to the nature of contemporary higher education in the UK. It should acknowledge that current higher education aims to be learner-centred and that excellent learning today more-often-than-not involves academic teaching staff working with expert learning support staff. It should be a ‘Learning and Teaching Excellence Framework’.

37         The second challenge is ensure that the (L)TEF is, and is seen to be, an open-minded exercise rather than, as many will suspect, an exercise to justify funnelling a greater share of the available resources towards those HEPs already richly endowed, whether or not they promote excellent learning. Here, the participation of student and international observers in the work of the expert panels will help sustain the credibility and integrity of the endeavour.

How should the proposed connection between fee level and teaching quality be managed? 

a. What should be the relationship between the (Learning and) Teaching Excellence Framework and fee level? 

38         Linking the (L)TEF to fee levels would be a mistake. For providers who are able to demonstrate their excellence in learning and teaching, public acknowledgement of their excellence through the Framework will enable them to attract higher numbers of well qualified students. For them that should be sufficient reward.

b. What are the benefits or risks of this approach to setting fees?

39         Linking fee-levels to excellence in learning and teaching, evaluated through an (L)TEF conducted with diligence and integrity could encourage the enhancement of learning and teaching among HEPs generally. Should a TEF approach be adopted rather than an (L)TEF, however, we think that the exercise will generate hostility on the part of HEPs and damage the global reputation of UK higher education.

30 October 2015