Academic Funding

What Can REF Teach Us About Impact?

October 7, 2014 1324

Ever since reporting on the Research Excellence Framework began, a number of debates and controversies have followed. The REF has seemingly presented a number of problems for how academics see their role and how their work is perceived, and valued, by their managers. The new rules of this research assessment instrument appeared allusive for many academics, whilst a number of concerns were raised in relation to how ‘impact’ would be measured and how the REF might shape behaviors within the wider higher education landscape.

LSE Impact logo

This article by Tony Murphy and Daniel Sage originally appeared on the LSE Impact of Social Sciences blog as “Perceptions and ‘impacts’ of the REF: Key aim for next round should be to explore apprehension and minimise anxieties” and is reposted under the Creative Commons license (CC BY 3.0).

As curious academics, we wanted to understand what was being written about the REF and what this might signal in terms of the broader ‘impacts’ of the REF: from individual academics all the way up to the nature of academia itself. In order to do this, we undertook an analysis of the media: collecting media outputs on the REF over a two-year period to identify who was writing about the REF, what issues were being discussed alongside it, and the extent to which reporting was positive or negative.

Our findings raised some interesting ideas about the way in which the REF has been experienced by different types of authors. The results are published in a paper for the Journal of Higher Education Policy and Management. Our first key finding was – perhaps unsurprisingly – that reporting on the REF tended to be negative, regardless of who wrote the article. However, articles by senior academics (most often professors) were the most negative. We also found differences in the level of negativity according to subject area; for example, authors from the natural sciences were significantly more negative than social science authors.

murphyref2

We also found that those based at research intensive institutions were more likely to be writing about the REF in the first place; thus suggesting that they are more likely to have their voices heard. As we neared the REF submissions deadline we found a greater portion of articles citing the REF.  However, as we neared the REF submissions date we also found that reporting on the REF became less negative on an aggregate level – still negative, but not quite as negative as earlier in the cycle. Had academics become more at ease with the process or, perhaps, were they more focused on the task at hand?

Amongst the prominent themes discussed in the articles, ‘impact’ was the most frequent one. Others included ‘funding’ and ‘marketization.’ We also found that ‘institutional inequalities,’ ‘staff and management tensions,’ ‘intellectual freedom,’ ‘dishonest practice,’ ‘staff morale,’ ‘recruitment’ and ‘pressures to publish’ also figured strongly. For each of these themes we found, on an aggregate level, that the discussions around them were negative. The discussions around ‘staff morale’ and ‘staff and management tensions’ were the most negative on our scale. The level of negativity varied according to author background, as well as the issues being discussed.

murphyref1

From this, we argue that the REF has seemingly been experienced by different authors in a number of varying ways; different issues seem to be of concern for different types of authors, and some issues seem to attract a more negative discussion than others. We were particularly interested in the high level of negativity apportioned to professors, and the data seems to link this to a concern for ‘intellectual freedom.’ A further interesting finding is the comparatively heightened concern for ‘impact’ among junior academics and those at ‘teaching intensive’ institutions.

In all, we suggest that such findings offer a sense of some of the issues associated with the REF; they signal the range of concerns academics have around the REF and its processes. The perception that the REF can adversely shape behaviors is particularly noteworthy; we found discussions around ‘intellectual freedom,’ ‘dishonest practice’ and ‘disregard for certain research.’ We argue that this makes it ever-more important that we get it right when thinking about ‘impact’ in research and the pressures academics face, and how this might manifest itself in dubious or even dishonest research practices.

Accountability and transparency are, most academics would agree, vital within the management and funding of universities. Naturally, this extends to research activities. However, future research assessment exercises must be as transparent as possible, with clearly defined and accessible rules. We must also consider the possibility of adverse or even perverse outcomes when defining those rules. Exactly how the next research assessment process will pan-out is hard to predict with any certainty at this point in time.  However, as our research suggests, a key aim of the process should be to explore the broader apprehension that academics approach it with.  And, where possible, strive to minimize such anxieties.

***

For further details please see the original paper: Murphy, T., Sage, D. (forthcoming,2014). “Perceptions of the UK’s Research Excellence Framework 2014: a media analysis.” Journal of Higher Education Policy and Management. 36(6).


Tony Murphy is senior lecturer in criminology within the Department of Law and Criminology at Sheffield Hallam University. Daniel Sage is a doctoral researcher at the University of Stirling. In January 2015 he will take up a post in the School of Media, Culture and Society at the University of West of Scotland.

View all posts by Tony Murphy and Daniel Sage

Related Articles

Research Assessment, Scientometrics, and Qualitative v. Quantitative Measures
Impact
September 23, 2024

Research Assessment, Scientometrics, and Qualitative v. Quantitative Measures

Read Now
Paper to Advance Debate on Dual-Process Theories Genuinely Advanced Debate
Impact
September 18, 2024

Paper to Advance Debate on Dual-Process Theories Genuinely Advanced Debate

Read Now
Webinar: Fundamentals of Research Impact
Event
September 4, 2024

Webinar: Fundamentals of Research Impact

Read Now
Paper Opening Science to the New Statistics Proves Its Import a Decade Later
Impact
July 2, 2024

Paper Opening Science to the New Statistics Proves Its Import a Decade Later

Read Now
A Milestone Dataset on the Road to Self-Driving Cars Proves Highly Popular

A Milestone Dataset on the Road to Self-Driving Cars Proves Highly Popular

The idea of an autonomous vehicle – i.e., a self-driving car – isn’t particularly new. Leonardo da Vinci had some ideas he […]

Read Now
Why Social Science? Because It Can Help Contribute to AI That Benefits Society

Why Social Science? Because It Can Help Contribute to AI That Benefits Society

Social sciences can also inform the design and creation of ethical frameworks and guidelines for AI development and for deployment into systems. Social scientists can contribute expertise: on data quality, equity, and reliability; on how bias manifests in AI algorithms and decision-making processes; on how AI technologies impact marginalized communities and exacerbate existing inequities; and on topics such as fairness, transparency, privacy, and accountability.

Read Now
Digital Scholarly Records are Facing New Risks

Digital Scholarly Records are Facing New Risks

Drawing on a study of Crossref DOI data, Martin Eve finds evidence to suggest that the current standard of digital preservation could fall worryingly short of ensuring persistent accurate record of scholarly works.

Read Now
0 0 votes
Article Rating
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Newest
Oldest Most Voted
Inline Feedbacks
View all comments