Impact ≠ Impact ≠ Impact 

Drops impact
There’s impact, and then there’s impact …

In this second response to Ziyad Marar’s thought piece “On Measuring Social Science Impact” from Organizational Studies, Anne-Wil Harzing, professor of international management and staff development lead at Middlesex University Business School, sets the stage for further discussion by defining impact in terms of progressing scientific knowledge, developing critical thinking, and addressing societal problems. 

It should be easy, shouldn’t it? Impact is impact is impact. Well, no… When talking about research impact, academics often have very different understandings of the concept. Worse still, they might not even be aware that different interpretations are possible.

As a result, any discussion about research impact soon descends into a Babylonian speech confusion. It also means that many academics are struggling to show the impact of their research, for instance when making a case for promotion (see: Academic promotion tips (1) – What makes a successful application). In this post, I will provide a brief clarification of the concept of research impact and show how it differs by academic role.  

What is impact? 

The Oxford dictionary defines impact as “a marked effect or influence.” Research impact thus means that our research has affected or influenced something or someone. Unfortunately, this immediately throws up even more questions:  

  • Whom has it impacted, i.e., who is the target audience?  
  • How has it made an impact, i.e., what was its ultimate goal?  
  • Through what means has this impact occurred, i.e., what are the primary outlets?  
  • How do we know this impact has occurred, i.e., how can we measure it?     

The answer to these four questions very much depends on the specific academic role we are looking at (see table). 

How research impact differs by academic role 

Academic role Whom? 
Target audience 
How?  
Ultimate goal
Through which means? 
Primary outlets 
How do we know?  
Measures 
Research Other academics Progress scientific knowledge Academic journals, Research Monographs, Conferences  Citations in academic journals / books / conference papers 
Teaching Students Develop critical thinking skills Textbooks, Practitioner journals Citations in textbooks / syllabi 
External 
Engagement 
Industry, Government, Public and society Address societal problems Practitioner journals / magazines, Policy reports, (Social) Media, Collaborative projects Citations in policy documents, policy / legal changes, improved HDI / SDG 
HDI = United Nations Human Development Index ; SDG = United Nations Sustainable Development Goals 

Research role 

In the research role of our academic jobs, our target audience is other academics, and our ultimate goal is to progress scientific knowledge by incorporating our work in the body of scholarly knowledge.  

We connect with other academics through academic journals, research monographs, and conferences. The preferred types of outlets are highly dependent on disciplinary norms and preferences. In the Life Sciences and Natural Sciences academic journals “rule.” In the Humanities and some of the Social Sciences, books remain a popular medium (see: Own your place in the world by writing a book). In some Engineering disciplines, conference papers are a popular way of diffusing knowledge quickly.  

Whether or not we have influenced other academics is typically measured by citations in academic journals, books, and conference papers. Although we know that academics are sometimes careless in their referencing (see: Are referencing errors undermining our scholarship and credibility?) and there are many reasons to cite papers (see On the shoulders of giants? Motives to cite in management research), one would normally expect citations to signify at least some level of impact on the citing academic. Citations can also be field normalized to account for differences in citation practices across disciplines (see: From h-index to hIa: The ins and outs of research metrics). 

Teaching role 

As academics, we all have a tremendous impact on our students. Obviously, some of this impact is unrelated to our research. However, in any good university, research feeds into the classroom and students benefit from research-informed teaching, allowing them to develop critical thinking skills.  

Graphic on P:erspectives on Social Science Impact
Click above to see other pieces of this series as they arrive and to read an excerpt from the essay “On Measuring Social Science Impact.”

As academics, we facilitate this directly through prescribing our own and other academics’ research as course readings. But in many cases, we will need to “translate” our research to make it more accessible for a student audience. We do so through publishing textbooks or articles in practitioner journals such as Harvard Business Review.  

So how do you know whether your research is used beyond your own classroom? You can find out if your research is cited in textbooks by using Google Books. To discover whether your publications (both academic articles and textbooks or practitioner articles) are listed in teaching syllabi, Open Syllabus Explorer (See Open Syllabus Explorer: evidencing research-based teaching?) is an incredibly useful tool. 

Engagement role 

So far, we have discussed the two key functions of any university: research and teaching. But there is also a third function: external engagement. This captures the impact of our research on industry, government, and the public/society at large, with the ultimate goal being to address key societal problems. It is the kind of impact incorporated in Impact Case Studies in the UK Research Excellence Framework (REF), defined as “an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment, or quality of life, beyond academia”. 

Making our research accessible to an audience outside academia typically means “translating” our research for non-academic use by writing articles for practitioner / professional journals and magazines, as well as publishing policy reports. It may involve the use of (social) media, not just to diffuse our already published work, but to allow continued engagement of non-academic audiences in our research. Integration of non-academic stakeholders from inception of research projects is increasingly common in the social sciences. 

Measuring this type of impact is challenging. Recent efforts like Overton – a database that captures policy impact by tracking citations in policy documents – might help. Academics may also be able to evidence changes in government policy or legislation that are linked to their research. A more general case for impact can be made by referring to frameworks such as the United Nations Human Development Index and Sustainable Development Goals.  

Horses for courses 

Different universities have very different strategies for research impact, especially in the research and engagement roles. In the 2021 REF, the national research evaluation in the UK, the field of Business and Management showed a very different ranking of universities depending on which of the constituent criteria you focused on. London Business School, London School of Economics, and University College London excelled in the perceived quality of their publications, but only had an average performance when looking at their societal impact. In contrast, Middlesex University, the University of Westminster – both modern universities – and SOAS (the School of Oriental and African Studies) topped the list for societal impact. However, they only had an average performance when looking at the perceived quality of their publications.  

In sum: measuring impact 

It is almost impossible to measure the ultimate effect for these three areas of research impact – progressing scientific knowledge, developing critical thinking, and addressing societal problems – accurately and comprehensively by using citations or other quantitative metrics alone. On the other hand, relying only on testimonials and a fully narrative approach to establishing research impact is unlikely to convince audiences, either. So in evidencing impact – for instance when making a case for promotion – academics are advised to rely on a combination of metrics, qualitative evidence such as testimonials, and career narratives.


Resources 

How to Improve your research profile, impact, and reputation 

How to write Effective promotion applications 

How to use Social media to support your career 

5 1 vote
Article Rating

Anne-Wil Harzing

Anne-Wil Harzing is professor of international management at Middlesex University, London and visiting professor of international management at Tilburg University. Her research interests include international human resource management, expatriate management, HQ-subsidiary relationships, cross-cultural management, transfer of HRM practices, the role of language in international business, the international research process, and the quality and impact of academic research. She is a fellow of the Academy of International Business and a founding member of CYGNA, a network of female scholars in the London area.

0
Would love your thoughts, please comment.x
()
x