News

Academy of Management Report on Measuring Scholarly Impact

March 13, 2018 1650

To lead development of valid and reliable measures of scholarship evaluation, the Academy of Management (AOM) recently published an empirical report on Measuring and Achieving Scholarly Impact. This post highlights the major findings, including pluralistic measures of impact, and wider audiences for research.

LSE-impact-blog-logo

This article by Usha Haley originally appeared on the LSE Impact of Social Sciences blog as “Beyond Impact Factors: an Academy of Management report on measuring scholarly impact” and is reposted under the Creative Commons license (CC BY 3.0).

With about 20,000 members worldwide, the AOM forms the preeminent scholarly association in management, publishing some of the most-reputed academic journals. The project had two interrelated parts: a qualitative study, and quantitative all-Academy survey on scholarly impact and its meaning to the AOM’s constituencies. This report includes the first survey of membership from a major academic association on what constitutes scholarly impact and which stakeholders have importance for scholarly research.

We began through open-ended, in-depth interviews with 30 members identified by the AOM’s Board of Governors as highly impactful (e.g. journal editors, AOM Presidents, and Fellows). The subsequent membership survey had a response rate of 19%. Despite their ubiquity as measures of scholarly impact, the majority of respondents (60%) indicated that journal rankings and lists, including Impact Factors, probably did not (20%), definitely did not (8%), or might or might not (32%) reflect scholarly impact. A minority (about 41%) indicated that rankings and lists definitely reflected (7%) or probably reflected (34%) scholarly impact.

Conversely, the top five indicators of scholarly impact on a five-point scale (1 = very unimportant, to 5 = very important) were:

  • Scholarly articles in top-tier journals – 4.49
  • Scholarly citations to research – 4.21
  • Scholarly books – 3.94
  • Competitive research grants – 3.93
  • Articles in practitioner-oriented/industry publications – 3.88.

Indicating the historical focus on internal audiences for academic research, on a five-point scale (1 = very unimportant, to 5 = very important), respondents saw the top five research audiences as:

  • Other academics in management – 4.48
  • Top management and decision-makers in companies – 4.26
  • Government and policymakers – 4.08
  • Other academics in the social sciences – 4.06
  • Students – 4.0.

Yet, about 54% of the survey’s respondents considered impact on practice as either strongly important (31%) or intensely important (23%); only 7% viewed impact on practice as not at all important as a component of scholarly impact.

Similarly, about 46% of survey respondents considered impact on government policy as either strongly important (27%) or intensely important (19%); only 10% viewed impact on government policy as not at all important as a component of scholarly impact.

Though more difficult to publish, about 59% of the respondents viewed interdisciplinary research, as probably more impactful (31%) or definitely more impactful (28%) than research that draws on one discipline; just 4% of the membership viewed interdisciplinary research as definitely not more impactful than research drawing on one discipline.

Respondents overwhelmingly saw institutional support as very strong for publications in top-tier journals, with other activities receiving far less, if any, support. On a five-point scale (1 = very unimportant, to 5 = very important), respondents ranked the top five beneficiaries of institutional support as:

  • Publications in top-tier journals – 4.54
  • Scholarly citations to research – 3.76
  • Obtaining research grants – 3.64
  • Published books – 3.07
  • Publications in practitioner journals – 2.84.

Most respondents (47%) saw their institutions as sometimes supporting their personal pursuit of scholarly impact. About 38% said their institution supported their pursuit almost every time (27%) or every time (11%). About 16% indicated their institutions almost never supported (13%) or never supported (3%) their pursuit of scholarly impact.

On the influence of the field, generally, the AOM’s membership thought that management research had been somewhat influential, but the greatest influence had been on other management academics including what they currently research and will research and teach. On a five-point scale (1 = very unimportant, to 5 = very important), the most influence was on:

  • Management theorising – 3.91
  • Teaching – 3.63
  • Future research practice – 3.59
  • Management policy and practice in large enterprises – 2.84
  • Students’ career decisions – 2.64.

The qualitative data showed persistent themes of high concern from senior scholars regarding the measures that institutions use to gauge scholarly impact, effects on career development, management research’s value, and societal benefits. Most of the scholars stated that the present system of faculty evaluation and business school rankings led to overreliance on more traditional techniques and methodologies, and what journal editors find acceptable. Some scholars identified that these developments had led to “junk science”, journals as “incestuous outlets for career-aspiring management academics”, underreliance on ideas, community and society, and excessive “balkanization” as management scholars became “angels dancing on a pin head”. Some raised concerns about the universal applicability and acceptance abroad of US faculty evaluation standards and research approaches that diminish scholarly impact. One scholar categorised the spread of US research standards globally as amounting to “imperialism” and a form of “colonialism”, with disregard for context.

Management scholars made several actionable recommendations on moving the field forward from its position of high academic legitimacy for wider social influence. Overall, to measure and to achieve scholarly impact, the report reinforced a need to develop composite measures of scholarly impact, to reduce excessive focus on methodologies and techniques, to increase value placed on developing ideas important to external constituencies, and to introduce more applications of theories to practice. As one scholar stated: “[support of this project] shows that the current Board is trying to fight the tradition of the status quo. The Academy has been so successful. Attendance at our annual meetings is the highest among any professional association. So, we have also become a victim of our own success, and there is little incentive to change. We are now criticised for our lack of relevance – and the Board sees that”.

In conclusion, the AOM report and findings fit into the larger conversation on research and researchers’ roles in society by exploring societal values and influence of research. The report has relevance for membership of the Academy, universities’ administrators, journal editors, tenure and promotion committees, accreditation agencies (e.g. AACSB, EQUIS), grant-giving organisations (e.g. NSF), policymakers, national reviews (e.g. PBRF, REF), and society at large.

Related Articles

Pandemic Nemesis: Illich reconsidered
News
June 14, 2024

Pandemic Nemesis: Illich reconsidered

Read Now
How ‘Dad Jokes’ Help Children Learn How To Handle Embarrassment
Insights
June 14, 2024

How ‘Dad Jokes’ Help Children Learn How To Handle Embarrassment

Read Now
How Social Science Can Hurt Those It Loves
Ethics
June 4, 2024

How Social Science Can Hurt Those It Loves

Read Now
Why Social Science? Because It Can Help Contribute to AI That Benefits Society
Industry
May 28, 2024

Why Social Science? Because It Can Help Contribute to AI That Benefits Society

Read Now
Digital Scholarly Records are Facing New Risks

Digital Scholarly Records are Facing New Risks

Drawing on a study of Crossref DOI data, Martin Eve finds evidence to suggest that the current standard of digital preservation could fall worryingly short of ensuring persistent accurate record of scholarly works.

Read Now
Biden Administration Releases ‘Blueprint’ For Using Social and Behavioral Science in Policy

Biden Administration Releases ‘Blueprint’ For Using Social and Behavioral Science in Policy

U.S. President Joseph Biden’s administration has laid down a marker buttressing the use of social and behavioral science in crafting policies for the federal government by releasing a 102-page Blueprint for the Use of Social and Behavioral Science to Advance Evidence-Based Policymaking.

Read Now
Analyzing the Impact: Social Media and Mental Health 

Analyzing the Impact: Social Media and Mental Health 

The social and behavioral sciences supply evidence-based research that enables us to make sense of the shifting online landscape pertaining to mental health. We’ll explore three freely accessible articles (listed below) that give us a fuller picture on how TikTok, Instagram, Snapchat, and online forums affect mental health. 

Read Now
0 0 votes
Article Rating
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Inline Feedbacks
View all comments

Subscribe to our mailing list

Get the latest news from the social and behavioral science community delivered straight to your inbox.

0
Would love your thoughts, please comment.x
()
x