Alex Bols, Deputy CEO, GuildHE
Originally delivered at the University of Chichester, 9th December 2015
Introduction
I would like to start by thanking you for inviting me to speak this evening, much of the education press and policy makers have been thinking of little else than how to develop a Teaching Excellence Framework for the last nine months and so this is a timely opportunity to think about this in more detail.
As has been mentioned I work at GuildHE which – alongside Universities UK – is one of the two officially recognised representative bodies in higher education, which basically means that I have been involved in vast numbers of meetings about the Green Paper and TEF for many months now. I’ll try and remember which bits I’m allowed to share and which bits are still confidential at this stage!
I must say that having been involved in discussing the detail for some time now preparing for this lecture has been both a very helpful, but also interesting, exercise bringing together and ordering all my thoughts on this topic.
I wanted to start with a bit of background about why the TEF is being developed and an overview of what’s being proposed, before going on to critique some of the key points, particularly around:
- The proposed metrics
- The link between TEF and fees
- And finally the timeline
I would like to end by suggesting a few principles for the TEF that might create a framework to be able to measure the effectiveness of the proposals in the future.
There will then be an opportunity for questions afterwards.
Why is the TEF being developed?
The TEF started its life as a partial sentence in the Conservative Party Manifesto back in April, inserted, or so we’ve been told, following lobbying by David Willetts, the former Universities Minister, who felt his reforms of the last Parliament – which resulted in increased tuition fees, more information for prospective students and new private providers – hadn’t driven up the quality of teaching as quickly as he would have wanted.
Indeed Jo Johnson specifically referred to the Willetts’ reforms and ambition in his first Ministerial speech back in July. This was the same speech he described some university teaching as “lamentable” that got him into trouble at the Select Committee yesterday. The Green Paper also refers to the HEA-HEPI Survey which showed that only 35% of students thought they had received ‘good’ or ‘very good’ value for money.
Having said that, there has also been a suggestion that perhaps it was “filler text” inserted into the Manifesto on the assumption that it could be negotiated away in the expected coalition talks. However, as we know the Election didn’t result in a coalition but rather a small Conservative majority. The size of the majority has many implications, not least it limits the ability of the Government to be able to pass legislation that is not specially referred to in the manifesto and so this meant that following the Election Ministers, and their officials, returned to the detail of what was in the manifesto with gusto.
However, whatever the gestation of the TEF we now have an ambitious Minister clearly on his way up, and looking for a quick promotion, and so wants to make an impact – and he sees the TEF as the vehicle to do this, now citing the TEF as one of his main priorities.
However, it’s also worth remembering that this isn’t the first time Government has tried to improve the quality of teaching – indeed even over the last 20 years since I’ve been involved in higher education policy we’ve been through the Teaching Quality Assessments of the late 1990s, the 2003 White Paper which resulted in the CETLs, National Teaching Fellowship Scheme, Teaching Quality Enhancement Fund and the National Student Survey as well as the Coalition Government’s belief that a market-driven approach would improve teaching.
Elusive nature of teaching excellence
If anything the number and diversity of initiatives over the past few years just demonstrates the real challenges of being able to drive change in this area at a national level, not least in terms of even being able to define what we mean by good teaching and how we can measure it, or indeed whether there are even proxies that are able to indicate good teaching.
When I think back to my own experiences, there are always a few excellent teachers that stand out, who really inspired me to engage with the subject, but equally if I try and put my finger on why they were so good and why I responded to them I can’t think of a particular trait that they all shared, or the reason why they managed to connect with my at a given time and others didn’t.
Indeed the Green Paper itself recognised that “There is no one broadly accepted definition of “teaching excellence”. In practice it has many interpretations and there are likely to be different ways of measuring it.” It also goes on to recognise that “not all students will achieve their best within the same model of teaching”. I will discuss the metrics in more detail later, but there is clearly the challenge of being able to separate out the teaching inputs, from the learning outcomes.
Focusing on excellent teaching suggests a teacher-led approach – which if we were looking at trying to measure this might include observations, contact hours, proportion of small-group work and looking at proxies such as teaching qualifications and CPD of staff or indeed the wider learning environment such as library resources or support to ensure that they complete – all of which could end up being fairly burdensome for institutions.
However, if we are trying to measure student learning, which the Minister, seemed to suggest in his July speech we would need to be looking at the much harder to measure area of student outcomes – such as the value-added by the institution, or “learning gain”, or their level of employability.
In reality we will probably be looking at a combination of all of these – although the question of institutional burden need to be considered – but this just highlights that it is such a shame that we have slipped into calling it a Teaching Excellence Framework. Not least because it immediately draws comparison to the REF and its cyclical review of university research, estimated to cost over a billion pounds.
It is probably too late to change the name – indeed I made a pitch for this at a conference shortly after the Election – but if not a Learning Excellence Framework then at least a Framework for Excellent Learning and Teaching (FELT) would better reflect what it is we’re trying to achieve and measure.
What’s being proposed
Before I move on critique the proposals it would probably be helpful just to give a brief summary of what is being proposed.
WonkHE – a blog site for those interested in higher education policy – have helpfully produced this model for the current proposals, although it is worth saying that the Green Paper is still relatively high level looking at the level of principle and suggesting some potential approaches, but there will be much more detail provided in the next stage of the process when BIS produce their technical consultation in the Spring following initial responses to the Green Paper.
Image created by Wonkhe. Click to enlarge.
The Minister has made clear that he wants the TEF reviews to take place in 2016/17 so that the outcomes of these will be in place for 2017/18 and that from the start a positive outcome in the TEF will enable institutions to receive benefits that are “both financial and reputational”. Which in the early phases of TEF is likely to mean that institutions will be able to increase their fees by inflation above the £9,000 cap.
Below the dotted orange line is what will be implemented from 2017/18 with institutions successfully passing a QAA Higher Education Review being able to increase their fees by inflation, there is a suggestion that this may remain as the TEF level 1 beyond year 1, and that the level 1 award would last for up to three years.
From year 2018/19 – with assessments taking place in 2017/18 – there will be a more granular approach with institutions able to apply for levels 2-4. These awards would also last for up to three years and will feed into any further fee cap and fee loan cap uplift.
There will be a technical consultation in the Spring to identify the detail of this but the initial proposals for this phase of the process focuses on three sets of metrics linked to teaching quality as demonstrated by the National Student Survey, employment data likely to be DLHE and student retention information. Alongside this data a short institutional narrative would set the data in context, which will then be reviewed by an independent panel of experts including students. Institutions will also have to meet some other conditional tests such as an OFFA access agreement, meeting the Competition and Markets Authority (CMA) guidance and the Minister has expressed his interest in institutions considering a Grade Point Average system.
The Minister has suggested that he wants different financial rewards to be made on the basis of different levels – although with inflation currently at -0.1 at the moment this doesn’t give much room for flexibility and hints to the concerns being expressed by some organisations that this may give rise to fee increases, or indeed decreases – with say a level 1 allowing institutions to charge up to £7,500, a level 2 up to £9,000, level 3 up to £11,000 and a level 4 up to £13,000, although it should be recognised quite how difficult this would be and would require a vote in the Parliament, and just thinking back to 2004 with Tony Blair and his majority of 169 still only just passing the HE Act by 5 votes.
It is then likely that the metrics involved will evolve over time to include a wider range of measures that better reflect teaching quality – I will come on this in more detail shortly – but before I move on I wanted to flag a particular concern in year 2 about the robustness of the judgements of the independent panel. The panel will be looking at the limited number of metrics that I mentioned earlier but it is unclear how they will be able to develop a way of differentiating between institutions to create three levels of award, and this is likely to result in legal challenge or lengthy appeals, especially if linked to financial return. Indeed I suggested to BIS earlier in the week that they may want to consider having the two highest levels as just being reputational benefits rather than financial in years 2 and 3, with the inflation linked to a TEF level 2 as a way of avoiding this legal challenge.
This is also likely to give rise to a new variant on the question of a whether a 2:1 from University X is comparable with a 2:1 from University Y – with the new version being how we know whether a Level 4 TEF at one institution is comparable to a level 4 TEF at another institution.
Learning the lessons from REF
So I have given a brief overview of where we are, and how we got here, I now wanted to focus on what I see as some of the key questions that we will need to ask as we move from the principle of introducing a TEF to the practical challenges of making it a reality.
The first point I wanted to make was about learning the lessons from the development of the RAE/REF. This has continued to evolve over more than 20 years and has taken time to get right, and so when we think about TEF we shouldn’t try and rush to get everything in place for year 1, the most important thing is not getting it wrong.
It will therefore be important that the TEF continues to evolve as more appropriate metrics and proxies become available. We don’t want to undermine the TEF by using metrics that are not seen as being proxies of excellence. It will also be important given the backdrop of the recent Chancellor’s Comprehensive Spending Review and the funding efficiencies and reductions that the sector will be expected to make that the TEF is simple, isn’t bureaucratic or burdensome.
Counting what’s measured, measuring what counts?
This then brings me on to my central theme, and the title of this evening’s talk, namely the metrics that the Government are proposing and whether we measure what counts, rather than simply counting what is already being measured.
As I highlighted earlier, there is no commonly agreed single definition of teaching excellence, domestically or internationally. Therefore rather than attempting to measure this through creating a hugely bureaucratic process – such as the QAA’s Teaching Quality Assessment of the late 1990s – it will be important that we use of proxies for teaching quality, whilst at the same time recognising the limitations and risks of using such proxies.
However, too great a focus on data and metrics risks confusing measuring excellence with assembling and publishing the least worst set of things that can easily be measured. This brings two key issues to mind. Firstly, a set of simple metrics will always struggle to reflect the real diversity across the sector and this must be taken into consideration when reflecting on the data. It will therefore be essential that data is effectively benchmarked for the type of subjects taught and the cohort of students studying are taken into consideration – just thinking about the NSS data there are real differences between different subject disciplines and the HEFCE Performance Indicators already take into consideration other factors that can distort data such as student background and geographical location.
The second factor to remember is that it is important to have a narrative provided by the institution so that they are able to consider the wider range of activities beyond the data and also that they are able to set the data on context. There have been two major reviews and pieces of research recently that have highlighted the flaws in relying solely on the data. Firstly, the recent independent report on the use of metrics in research assessment (the so-called Wilsdon report) concluded that quantitative metrics are no substitute for academic peer review. I believe that similar considerations apply to the use of teaching-related metrics and this was reinforced by the recent research at King’s College London on the inability of data to accurately predict institutional performance in QAA reviews.
This raises the importance that the institutional statement should be given as much weighting as the data, rather than seeing the data as the entry point for different TEF levels. Indeed the institutional statement should be seen as a robust evidence base on which to make a judgement rather than being seen as a way of justifying the data. This also raises an important point about ensuring that the Independent Panel is made up of people who have an understanding of universities and the institutional context and are able to interpret the qualitative statement rather than just over-relying on the data.
Data
In the early stages of the TEF, in order to meet the tight timescale, it will be important to use existing measures that have developed over time and are considered robust enough to make the type of (funding) decisions that I referred to earlier.
It is likely that whatever data is used in the early years may not be as robust as potential future metrics but this would suggest that BIS should not try and make the data do things that it is not able to withstand – and in particular concerns surrounding the ability of three metrics and a short institutional narrative to be able to result in three levels of differentiation in the outcomes of TEF should be seriously questioned.
As highlighted in the model the three data sources cited in the Green Paper are NSS, DLHE and retention data.
The biggest area of concern that I have heard across the sector and also during the recent BIS Select Committee Inquiry hearings related to the use of employment data. There are real questions about how this relates to teaching excellence and whether it is actually a proxy for the outcomes of the learning or rather more likely to reflect other factors such as course and career choice as well as other factors such as social capital.
However, whilst recognising these challenges it should also be noted that students have been increasingly citing the importance of employability and job prospects when applying to university – and students clearly consider it, at least in part, as a measure for their time at university – and so it is right that this be something considered – it is at least better than trying to rely on salary data which has been suggested by some. It will however be important to use more than just the six-month DLHE data which is skewed by issues such as students doing unpaid internships or having a portfolio of careers in some professions such as the creative industries.
Almost as concerning as the use of employability data is use of retention data. It is, of course, important to recognise student progression and retention can be a proxy for the support that students receive and the wider learning environment. However it is important that this doesn’t become a long-term metric, especially if TEF is linked to increased fees, as it could create unintended consequences based on the fact that students from particular backgrounds are less likely to drop-out. It is also worth reflecting that some students “drop-out” for the right reasons – whether moving to a different institution, realising they’ve made a mistake or indeed receiving a job offer – and we wouldn’t want to incentivise them staying on their course if it’s not right for them.
The NSS data is likely to be the questions relating to teaching quality rather than the overall question about satisfaction – which is clearly a positive step but there are still concerns about the data. The questions relating to teaching quality have developed over ten years and there is tracking data to measure trends, and institutions have been using it for enhancement, and there is at least a case to be made that it could be a proxy for teaching quality. However, there will of course be concerns about students making judgements when – in most cases – they have nothing to compare their experience to, as well as whether they are reflecting on teaching or their own learning.
All in all I would suggest that whilst there are flaws with the proposed metrics they do at least use existing data that is understood and are universal and apply to all institutions, and so I can probably live with them for a couple of years – but I do want to set down a marker that if the Minister moves role in a couple of years and a new Minister comes in and decides TEF 2 is too complex that we ensure we aren’t left with these metrics for many years to come.
Possible future metrics
Having critiqued the current metrics I suppose I should at least suggest a couple of possible future measures.
There have been many proxies suggested but three broad areas surrounding teaching quality, learning environments and student outcomes feels like the right starting point for discussing the possible bundle of future metrics. Indeed it is worth reflecting on Graham Gibbs’s Dimensions of Quality research that have done so much to shape Government thinking in this area.
There could be various proxies in these categories including teaching qualifications and ongoing CPD, questions around how engaged students are with their learning that HEFCE are consulting on revising the NSS, and at a future point measures of “value added or learning gain” which are currently being piloted.
It may also be attractive for the Government to consider looking at staff/student ratios, investment in teaching and contact hours although any future metrics must both pass the test of not introducing unintended consequences and also not creating an overly bureaucratic system.
However as highlighted earlier, my main point is that we need to recognise the diversity of the sector and subject and so shouldn’t just rely on data but build a wider picture using context-specific information.
Gaming the system
Finally it will be important when considering how to create a new system how be minimise the ability of institutions that might be tempted to try and game the system. This will be especially important when there are funding and reputational implications it is likely that institutions will try and present their institution in the best possible light.
Diversity & Institution versus subject/department level
It is important to recognise that teaching excellence is dependent on context: institutional mission, cohort of learners and subject. Use of these data would need to be benchmarked to ensure you were comparing like with like. Crucially, an effective TEF cannot be developed on a ‘one size fits all’ basis – this is something that HEFCE have been trying to move away from in the context of quality assessment and we shouldn’t slip back into it through TEF. It has to be sensitive to context – able to reflect institutional diversity and the different subject disciplines and pedagogies.
As we think about how a TEF might be developed it is worth saying that a culture of developing outstanding teaching is both supported and facilitated at the institutional level, but delivered at the departmental level and so any future TEF will need to consider both dimensions.
The institutional level should demonstrate the wider strategies, policies and practices as well the institution’s reward and recognition processes. However to establish the excellence of the teaching it will need to be done at the departmental level, even in the best institutions there will be variation in the quality of the teaching across departments. These differences within an institution may highlight different quality but also the different approaches in different disciplines as well as the backgrounds of the students themselves.
The link between TEF and fees
The second question that I wanted to address is the proposed linking the outcome of TEF to being able to raise fees – whether by inflation in the first instance, or reconsidering the fee cap itself as TEF develops.
NUS and others have rightly suggested that students should already be receiving excellent teaching, at the recent evidence session at the Select Committee NUS President Megan Dunn said “If we are saying to students that you can pay £9,000 and not get excellent teaching, then there is a problem.” Going on to describe the link to fees as “dangerous”.
In addition to the expectation that students should be receiving excellent teaching anyway there has also been concern – as touched on earlier – that the financial implications could have an impact on how institutions engage with the process, from a focus on enhancing teaching to creating too much incentive to focus on the metrics and data that inform the assessment and so could result in playing the system. Or indeed the perverse scenario where students decide to mark down the NSS to ensure lower fees for future students.
I would suggest that the reputational impact of an effective Teaching Excellence Framework is likely to drive institutional behaviour almost as much as any link to fee level. This is already demonstrated by the huge international interest in the proposals. There is therefore a question as to whether the fee link is needed.
It almost feels as though it is a rather blunt way of introducing the market into fee levels that the Government briefly expected when they introduced £9,000 fees. Following the race to £9k it could be suggested that the different TEF levels will be a way in introducing real market differentiation to prospective students – so that they are able to judge value for money based on the different fees levels, and all the difficulties that arise from that.
In a potential future scenario where there could be much greater fee differentiation it will be important to consider the impact that whilst the “best” teaching is rewarded with increased funding, there may be a scenario that less good teaching is penalised and so doesn’t have the ability to invest to ensure its improvement. Or indeed given the time lag in data, it may not recognise improvements already made. This will either result in a degradation of quality of teaching for these students or could result is closure of courses which will impact on student choice, especially those not able to travel to other institutions.
However, having critiqued the fee link it is perhaps just worth mentioning that as I look back at what has actually resulted in changing institutional behaviour over many years it is the financial incentive that often plays a central role. It is likely that the practical impact of a fee link will ensure that there is a renewed focus on the quality across the higher education sector and could result in the intended re-balancing of the emphasis between research and teaching that the Minister has highlighted as being central to the proposals.
The link to teaching excellence also ensures that institutions are able to increase their fees to recognise both inflation and the real terms cut in fees since they were introduced and the rising costs facing universities. It is important to have a process that enables institutions to raise their fees by inflation, and if you are going to do that then surely linking it to improving quality at least seems the most defensible approach. There is also the question that if not linked to quality what alternative would the sector suggest to enable institutions to raise fees by inflation – and that would satisfy the Chancellor.
Timescale
Moving on it to think of other wider questions associated with the TEF, when I think about all the other challenges of the TEF it is when I think about the timescale that I become most nervous.
As ever when Ministers have a “good idea” they want to see it implemented as soon as possible – perhaps because they want to leave a legacy, perhaps because they want to introduce it before they move jobs and their successor starts to tinker with it or perhaps just because they’re not so involved in the detail that they don’t know how long these things actually take to get right.
It was suggested at the Select Committee last week that the timescale is achievable but only because the scope is fairly limited in the first couple of years and so we have substituted the speed of implementation for the meaningfulness of the outcome.
It was good that BIS listened to the sector that introducing something new in time for 2017/18 would be impossible and so they are using a successful QAA Review as a proxy in the first year of TEF. But even looking at 2018/19 the timescale looks tight.
Just working it through, students starting in 2018/19 will begin the applications process in October 2017 (for Oxford, Cambridge, medicine, dentistry and veterinary courses), and January 2018 for other courses and institutions. Institutions successful in TEF will be able to raise their fees by inflation and so students will need to know the fee level as part of their decision-making. This means that institutions will need to update their websites by early Summer 2017, having already gone through their University Board of Governors for approval. Therefore decisions – including using independent panels – will need to be made by late Spring 2017.
But it is not just the speed that it concerning it is also the lack of piloting and so BIS will be creating a process without knowing whether it will work or what the unintended consequences might be – and when you consider the likely financial consequences this could also result in lengthy judicial review and appeals.
Staff and students
Before I think about wider principles of how we might judge the effectiveness of the TEF I wanted to briefly touch on a couple of other points that are worth considering:
Firstly the clear link between well-supported, paid and developed staff and teaching excellence. Whilst this hasn’t received much attention so far in the discussions surrounding the TEF it has been obliquely referenced through the recognition in the discussions surrounding the TEF about the link between qualified staff and their ongoing professional development with the quality of teaching. It is likely that this might form part of the basket of metrics as TEF evolves, but it does feel like this wider issue of the support that staff receive hasn’t been given sufficient emphasis.
In her evidence to the BIS Select Committee, Sally Hunt from UCU raised concerns about zero-hours contracts and casualization and the impact that this can have on the quality of teaching. She identified key pressures upon quality arising from casualisation including inadequate time for marking assessment, inability of teachers to access training and professional development, reduced student access to experienced academic staff, and the poor morale and the high turnover of staff.
Sally expressed her desire to see a new assessment system to examine the relationship between casualised employment arrangements and the quality of the student learning experience, for example, its impact on marking and assessment processes and the opportunities for staff on casual contracts to access professional development. Indeed the Chair of GuildHE, Professor Joy Carter ended her evidence to the Select Committee by highlighting the important role of satisfied and engaged staff in order to deliver high quality teaching and therefore the role of reward and recognition will be key to the TEF as it develops.
We should however recognise that in many GuildHE institutions there are many staff that work part-time and are also professionals in industry – this professional experience is particularly valued by students and so we must ensure this isn’t unintentionally caught up in this.
In addition to the impact of the security of staff contracts and reward and recognition issues it is also important to recognise the impact of how students are engaged in the process. I have already touched on the importance of future TEF metrics including how students are engaged in their own learning but we should also consider how students are involved in the evaluation of teaching excellence.
The views of students will be embedded into the TEF metrics through the use of the National Student Survey. But in recent years there has been much discussion about student engagement and moving towards students being recognised as partners in their education. In this scenario simply using student feedback through surveys feels like a fairly passive approach to engaging students. Whilst it isn’t yet exactly clear what TEF evaluations at a subject level will look like we must ensure that students are embedded into these processes.
Whilst we’re just looking at the institutional level, if there is an institutional narrative sitting alongside the data we should consider how student views are reflected as part of this, which could include it being a joint submission, a students’ union commentary or just ensuring that the statement is signed off by a university committee that includes student representation. This principle of student representation should also be an integral part of the independent peer review that will be reviewing the data and institutional statements.
We have already seen in recent years the benefit that students have brought to QA Review panels and as we move into a new regulatory landscape with an Office for Students, this wider – and deeper – engagement of students should be embedded in the process from the beginning.
How will we know if it was a success?
I suppose one of the key questions to consider is looking back in ten years’ time, how will we judge whether the TEF has been a success?
The Minister has emphasised a number of key objectives of the TEF in his speech 1st July 2015 particularly focusing on the outcomes of the TEF such as ensuring all students receive an excellent teaching experience that encourages original thinking, drives up engagement and prepares them for the world of work and a wider context of building a culture where teaching has equal status with research. But there will be as many challenges in trying to measure the impact of the TEF in relation to these objectives as there are trying to develop the right set of metrics in the first place.
This perhaps suggests a wider set of principles that are seen as underpinning the success of the higher education system. In his evidence to the BIS Select Committee Professor David Maguire the Vice Chancellor at the University of Greenwich highlighted five key principles that should be at the heart of TEF. “The first one is a focus on enhancement and getting better. The second one is about reputation. We need to ensure we seek to enhance and certainly do not weaken the reputation of our world-class sector. We need to preserve diversity. There is not one model of teaching excellence or of success. We need to recognise the different missions and different goals of different areas. Our sector is founded upon autonomy and the autonomous nature of institutions. We need to preserve that. We also want something that will be efficient. We do not want to spend a lot of money and time and deflect people from doing research and teaching by concentrating on filling in forms and delivering information.”
These five principles of enhancement, reputation, diversity of teaching and diversity of institution, autonomy and efficiency could be a helpful framework by which to measure the success of the TEF.
Conclusion
In conclusion the Teaching Excellence Framework provides a real opportunity to enhance teaching within universities and raise the profile of good teachers but there are clearly many potential pitfalls along the way that we need to avoid.
To download this lecture, click here: