Earlier this year, the results of the Research Excellence Framework were published, a 5-yearly attempt to grade and rank (from best to worst) universities throughout the UK in terms of their research excellence.  It is a far from perfect process, estimated to cost over £1billion, and is highly criticised throughout the academic world.  However, it’s important, because the rankings are used to inform funding decisions – the implicit promise being that the higher up the rankings you are, the more funding you get.

You can download the full REF 2014 results here.

You will see that the results are split by University, then by department, and then finally into Outputs, Impact and Environment.  The definition of these three can be seen here, but for the purposes of this blog post, we will focus only on Outputs – that is the traditional outputs of academic research (e.g. in scientific areas, these would be peer-reviewed papers).

I’ve cleaned up the Scottish data, summarised over all departments, and you can see it here.

Now, Times Higher Education describe two metrics which can be calculated on the data:

  • GPA (grade point average). Loosely speaking this can be thought of as the “rate” of high quality research.  The higher the value of the GPA, the more likely research at that institution will be of high quality.
  • Research Power.  This can be thought of as the volume of high quality research, and is simply the GPA multiplied by the FTE count.

The reason we have both of these is to separate institutions who submit only a very small amount of high quality research to the REF, from those that submit a larger amount of of mixed quality research.

Remember here we are only looking at research outputs.  So how do the Scottish Universities compare to one another?  Here are the Scottish institutions arranged by GPA:


And here are the Scottish institution arranged by Research Power:

scot14_rp_crop I don’t think there are any real surprises in these tables, if I’m honest.

Now, it is the Scottish Funding Council (SFC) in Scotland that distributes Government funds to universities in Scotland, and we can see the “Research Excellence” settlement for the next 4 years here.  You can download a sanitised version here.

If we take the 14/15 research excellence settlement as the baseline, then we can plot it against both GPA and Research Power: ref_correlate

Whilst in general, universities with a higher GPA generally receive more funding, this is normalised by research volume, and we can see clearly that research funding correlates with research power.

Those of you who have looked at the SFC funding figures will see that they change over the next few years.  So let’s take a look at those changes, using this year as a baseline. Here are the raw figures:



Does there look like there’s an outlier to you?  That data point in the bottom right is the University of Edinburgh – ranked first for research power in Scotland and second for GPA in Scotland; also ranked first for loss of funding.  However, Edinburgh receives the most funding from SFC, so let’s look at percentage drop in funding:


This changes the picture a bit: the biggest loss in funding in percentage terms is Robert Gordon University, losing 59% of its 2014 budget over the next 3 years.  Second worst off is the University of Edinburgh, losing 20%; then Dundee (13%); Aberdeen (10%); and St Andrews (4%).

In fact, 4 of the top 6 Scottish universities (ranked by GPA or by Research Power) are receiving a funding cut.  In contrast, 7 of the bottom 8 Scottish Universities (ranked by GPA or by Research Power) are receiving a funding increase.

It is hard to see this as anything other than the defunding of research excellence in Scotland.  The argument is that by taking money from “the rich” and giving it to “the poor”, the SFC will raise the quality of research at some of the lower ranked universities.  Well it might, but there is no guarantee.  At the same time, the opposite will occur – the quality of research at the higher ranked universities will drop.  By my calculations, the University of Edinburgh will lose £17m over the next 3 years.  That can only have a negative effect on research excellence.

I am not against Robin Hood politics – there absolutely should be a continuous re-distribution of wealth from the rich to the poor.  But does this work in a research setting?  The REF ranks universities on “research excellence”.  Universities strive to be as close to the top of those rankings as they possibly can be, and they spend a fortune to do it.  What point is the ranking if the result is that the best are worse off than they were before?  Put another way, could Edinburgh, Dundee, Aberdeen and St Andrews have done anything differently?  They all improved since 2008; they all came in the top 6 in Scotland.  Yet the reward is a cut in funding.  Perversely, had their research quality plummeted and had they been ranked in the bottom 8, would they have received a funding increase?

More likely is that the SFC intended to redistribute funding no matter what the outcome of the REF (note the SFC funding above is called the “research excellence grant”).  If that is the longer term policy of the SFC, that gradually funding will be moved from the higher ranked universities to the lower ranked, then they immediately remove the incentive for the higher ranked universities to improve or even maintain their ranking.  It could be argued that there is also little incentive for the lower ranked universities to improve, as they will get increased funding anyway.


Overall, I have to say, I am left completely bemused by the whole REF process.  What the **** was the point?