This blog was developed from the presentation by Dr Alex Bols, Deputy CEO GuildHE, at the Inside Government TEF Reset Conference 2022 on 30 June 2022

What does the new TEF mean for Small and Specialist Providers?

• Whether the new TEF will enable smaller and specialist institutions to demonstrate their excellence
• The impact of smaller – and therefore more volatile – datasets
• How students can be engaged where there are less formal students’ union structures

GuildHE is the representative body of smaller and specialist universities and colleges, representing 55 providers with a diverse range of specialisms from agriculture and land-based studies to creative and performing arts and the built environment to specialist health providers.

There are some very specific issues relating to smaller institutions, with smaller cohort sizes and their more volatile data as well as smaller students’ unions who may find it harder to develop student submissions. Secondly many smaller and specialist institutions did very well last time round, and so it is interesting to reflect on why that was and also whether it is likely to be repeated following the reforms.

The aim of the TEF has been to assess excellence in teaching at higher education providers and assess how they ensure excellent outcomes for their students in terms of graduate level employment or further study. Indeed the TEF Chair Chris Husbands suggested in a recent blog that “the TEF was designed to identify and celebrate outstanding outcomes from teaching across higher education and to serve as a tool for driving further quality improvement.”

Whether or not it actually achieves those aims – particularly when looking at institution level data – I’ll leave to one side as much has been said and written about that but I wanted to draw out some of the key challenges for smaller institutions and how TEF can assess their excellence.

The recruitment for TEF assessors is currently underway and it is outlined that the Assessment part will be “early/spring 2023” and assessment part two will be “spring to autumn 2023”. This would suggest a provisional timeline of students and providers to make their submissions, with a deadline in early 2023. Assessment of submissions would subsequently be carried out, with the outcomes available in time to inform prospective students for the 2024-25 recruitment cycle. So we are therefore likely to see the consultation response and start of the exercise over the Summer.

It is worth reflecting on TEF 1.0 and looking at the results of the first outcomes of TEF back in June 2017. Many specialist institutions received a Gold award including a number of very small and specialist institutions and it helped to showcase this excellence which wasn’t being highlighted through traditional league tables which are still heavily weighted towards research power, entry tariffs or even just don’t include many of the smallest institutions based on their size.

TEF was finally seen as a way of rectifying that. Looking back at whether that has actually been the case and the extent to which those institutions did well in TEF did well in recruitment is perhaps less clearly aligned and I will reflect on this later.

When the first TEF results were announced last time around only 59 institutions of the 295 providers that received the TEF awards received a Gold Award – and 46% of those were small or specialist higher education providers including 9 FE colleges delivering FE.

Why is it that smaller and specialist institutions did so well last time round? The obvious answer would be that in larger institutions there is more likely to be more variation in provision across the institution, with one or two subject areas being weaker – or just having a bad year in the NSS – having the potential to bring down the whole institution. Whereas for smaller – and particularly for specialist – institutions they are almost doing a subject-specific TEF.

When thinking about the next TEF, what are the key areas that have changed?

As an over-arching point – it is worth mentioning that as the quality systems in the four nations of the UK become ever more divergent I do wonder how attractive an approach to measuring Teaching Excellence that is primarily based on the English regulatory and quality system will be across the UK? Last time around a number of Scottish and Welsh institutions entered TEF, although interestingly no Northern Irish institutions. I have wondered whether this time more Scottish and Welsh institutions are likely to follow the Northern Irish lead and not opt-in to a process based on the English regulatory system, and what impact that might have in an international context and further emphasising the divergence of systems across the four nations.

Firstly, there is clearly the reduced emphasis on the National Student Survey in the TEF compared to the first iteration, but this has been a trend in earlier iterations.

Secondly, while the removal of the flags is probably a good thing I do have a sneaking concern about the scatter charts and how well they will be understood by panel members, many of whom will not have experience of using data in this way.

It is therefore vitally important that high-quality training on statistical modelling and outcomes should be provided to the panel in advance of seeing any real data. This is a particular worry for smaller providers where the statistical variability of the data may have more opportunity to be misinterpreted by panel members and it will therefore be harder for panel members to have confidence in our data presented to them. It will also be important for training of panel members to consider how they might perceive providers that have no data available by split metrics.

Equal weight for narrative

Probably the key change is around there being no “initial hypothesis” resulting from the data and that the data and narrative document will be given equal weighting. Firstly, giving more weight to the narrative provided by HEIs will enable all providers regardless of size or specialism to put together a strong submission and be judged fairly. The addition of a template will further ensure equity across the sector and support smaller providers, especially those who do not have the same level of resource to coordinate a provider submission.

We believe that the TEF process should be based on a fair and equal weighting of data and additional narrative provided by the institution and student submissions. This is why we hope the guidance genuinely assigns an equal weighting to the provider and student submission document.

Require improvement rating

Having been a school governor for 16 years and going through 6 OfSTEDs with different schools as chair of governors I would just say that the Requires Improvement wording has a very specific meaning in educational contexts and that is not appropriate for an assessment that is supposed to be above the quality minimum threshold. I can see the benefit of having a fourth rating – not least to show Bronze as a positive achievement – but would consider something like “Satisfactory” or “Meeting Quality thresholds” to be a better approach.

Data volatility in small institutions

I’ve already touched on some of these issues but just to reiterate that there will be a large number of HEIs that won’t necessarily have enough data for their split metrics to be published in all areas. It will be important for the guidance to provide greater clarity as to how the panel would form conclusions where data wasn’t available – would the narrative statement have more weight or would the panel ignore the gaps in data altogether?

Furthermore, where the data has a very low confidence interval (below 95%) how would the panel take this into account without unduly penalising smaller providers who by their very nature will have more volatile data?

The second point I wanted to mention that particularly impacts on smaller institutions with smaller data planning teams is that the data OfS has is very different to the provider/HESA record so understanding that can be frustrating especially if you have a small registrar team.

Interim assessments

One of the other changes in the new TEF is that it is proposed to run on a four-year cycle with no interim assessments. Whilst I recognise the benefit of not having the burden of have a permanent panel undertaking annual assessments, it does result in two key challenges.

Firstly, for institutions it means you’re stuck with a judgement once it’s been reached – which as an aside is likely to mean more legal challenges where an institution thinks that it deserves a higher rating. As a board member of an institution that was able to upgrade itself from Bronze to Silver last time round, it was both rewarding for institutions to recognise their teaching and learning, and in the case of that particular institution meant that they were able to move on from a poor set of NSS, which in a small institution is much more likely to fluctuate year on year.

Secondly, working with smaller institutions and also institutions that are still joining the OfS Register it means that institutions that pass the 500 student threshold or register with the OfS after the TEF cut-off period would be excluded from the exercise and having to wait four years till the next one. We believe that it would be worthwhile allowing a mid-cycle assessments just for those institutions that weren’t eligible at the cut-off point. This is especially true for providers seeking to join the OfS register when the speed at which they are registered is outside of their control.

Student submissions

GuildHE are strong advocates for student engagement and welcome the OfS taking on board our feedback from previous consultations that a student submission would add additional richness to the TEF process.

However, whilst our institutions are passionate about student engagement, due to their size this engagement is often a lot less formal than in larger institutions. It will therefore be important for panels to be mindful of the constraints for some student bodies to put together such submissions. GuildHE will support our student representatives with training and advice, but not all HEIs have students’ unions or resources to undertake this task. The template will be a helpful addition to support students to undertake an independent submission but panels should not judge the quality of the provider (nor the quality of the relationship between staff and students) solely on the quality of the student submission. We also welcome the proposals that students’ representatives can be more creative in how they present the student voice, rather than just assuming that this will happen through a formal written submission.

We welcome proposals that mean the timetable will be pushed back but with many of our students’ unions not having full time sabbatical officers or staff it will be also important to consider the timing in relation to winter exam and assessment periods.

What might a successful TEF look like in late 2023?

Criterion referencing Gold awards

We hope there will be fantastic outcomes for all the smaller and specialist institutions again. It will be important to ensure that any institution that meet the criteria for a Gold Award should be able to get one. If we are to truly showcase the excellence of English higher education then it wouldn’t be helpful to have minimum/maximum numbers of providers receiving Gold-Silver-Bronze. A norm-based approach based on a perceived Bell Curve would not be appropriate, although I can of course see the political motivations for such an approach. However, particularly in the context of this reformed TEF where enhancement and continual improvement is the cornerstone of the exercise, stifling the number of Gold awards for outstanding practice would be demotivating.