A Quick Examination of Existing Academic Impact Metrics and Concerns in Business Education

A new white paper from SAGE Business examines existing bibliometrics and institutional reward structures at play within business schools. We aim to move the dial toward ways in which societal impact could become central to the assessment of business and management research.

Usha Haley, left, and Andrew Jack

In a pioneering survey of its membership in 2017, the Academy of Management asked its worldwide membership what constituted scholarly impact. (With about 20,000 members worldwide, the AOM is the preeminent scholarly association in management. The subsequent membership survey had a response rate of 19 percent, or 700 respondents.)

Despite the ubiquity of journal impact factors as measures of scholarly impact, the majority of respondents (60 percent) indicated that journal rankings and lists, including impact factors, probably or definitely did not or might or might not reflect scholarly impact. Conversely, the top five indicators of scholarly impact were: Scholarly articles in top-tier journals, scholarly citations to research, scholarly books, competitive research grants, and articles in practitioner-oriented/industry publications.

Indicating the historical focus on internal audiences for academic research, respondents to the AOM survey saw the top five research audiences as other academics in management, top management and decision-makers in companies, government and policymakers, other academics in the social sciences, and students.

This is the first of seven posts excerpting and adapting the SAGE Business white paper “Measuring Societal Impact in Business & Management Research: From Challenges to Change.” The white paper includes a core section, written by Usha Haley and Andrew Jack, and complementary essays penned by Ben McLeish and Mike Taylor of Altmetric / Digital Science, Sir Cary L. Cooper at the Alliance Manchester Business School, Renate E. Meyer, WU Vienna / Copenhagen Business School Maura L. Scott, Florida State University.

On the influence of the field, generally, respondents thought that management research had been somewhat influential, but the greatest influence had been on other management academics, including what they currently research and will research and teach. Yet, about 54 percent of the survey’s respondents considered impact on practice as either strongly or intensely important. Similarly, about 46 percent considered impact on government policy as either strongly or intensely important. Though more difficult to publish, about 59 percent viewed interdisciplinary research as probably more or definitely more impactful than research that draws on one discipline. Respondents overwhelmingly saw institutional support as very strong for publications in top-tier journals, with other activities receiving far less, if any, support. Still, only 38 percent said their own institutions supported research with external impact.

Similarly, a global survey undertaken by SAGE in 2021 of social science academics (the first author was involved in drafting this survey and analyzing results), found that of the 373 global respondents in business and management (most from the United States, United Kingdom and India), 81 percent said that it was either important or highly important that their research have value outside academics, and 62 percent stated that their peers felt the same way about their own research. 

Two bar charts, the first showing the high Importance to me of my research having an effect outside academics (n=373 business academics, and the second, that academics strongly agree that Ultimately the goal of my research is to make a positive impact on society
Two bar charts, the first showing How my institution rewards efforts to apply research outside academia n=373 business academics), and the second the high Importance of research getting published in highly cited journals/journals with high impact factors (as
measured by Journal Citation Reports)

Ninety-four percent of the respondents either agreed or strongly agreed that ultimately the goal of their research is to make a positive impact on society. Yet, when asked if their institutions rewarded these efforts to apply research outside academia, only about a third thought they did so. Only 36 percent said that having external impact mattered for tenure, 34 percent said for awards, 32 percent said for funding further research, and, 30 percent for other research resources. Thirty-one percent of the respondents said their institutions provided no rewards or acknowledgments for having external impact through their research. Unsurprisingly, 63 percent either agreed or strongly agreed that ultimately the goal of their research is career advancement. Seventy-seven percent of the respondents stated that publishing their research in a highly cited journal with a high impact factor was either important or very important.

A 2022 poll of 9,609 academics by Times Higher Education found similar results. Aside from personal interactions with the researchers, the perceived quality of the journals where the researchers published their findings constituted the most influential factor when forming opinions of academic standing, according to 49 percent of the respondents. Citation metrics had lower support. Only 24 percent of respondents said a scholar’s h-index and other similar measures were important, and only 5 percent said they constituted the most crucial factor. These findings regarding citations could arise because of increasing awareness of gaming the system through self-citations, citations that speak negatively of the articles, forced citations by journal editors and reviewers, and citations made by researchers who haven’t even read the articles (see Haley, 2021). Additionally, citation counts vary by source—for instance, those in Google Scholar differ from those in Clarivate.

Few measures have exerted greater influence than the journal impact factor, or JIF. Initially developed to help librarians to purchase journals, Clarivate’s JIF has transmogrified into an evaluation of the quality of individual publications and of individual researchers.

The measure has come under extensive scrutiny and criticism: as an inaccurate estimate of citations of any article within a specific journal, as easy to manipulate, and with no associations to objective measures of quality. A decade ago, the San Francisco Declaration on Research Assessment (DORA), critiqued using impact factors as a surrogate measure of quality for individual articles and researchers. DORA currently has 20,000 individual and 2,600 institutional signatories worldwide.

The AOM survey and FT awards (see Jack, 2022) have identified interdisciplinary research as more impactful than single discipline research as it incorporates diverse perspectives and experiences. Yet, interdisciplinary research is also more difficult to publish. Ricardo Fini, Julien Jourdan, Markus Perkmann, and Laura Toschi argued last year in Organization Science that interdisciplinary researchers and research may threaten disciplines’ and evaluators’ distinctiveness and knowledge domains, and hence, evaluators may penalize them. High-performing, interdisciplinary researchers appeared to suffer the greatest penalties in small and distinctive academic discipline where evaluators appeared as representative members of their disciplines. Fini et al. argued that attempts to maintain social boundaries contributed to the relative lack of interdisciplinary research.

Upcoming Articles in This Series

“A Quick Examination of Existing Academic Impact Metrics and Concerns in Business Education” | Usha Haley and Andrew Jack

“How Might Societal Impact be Recognized within an FT Top 50 Journal?” | Renate E. Meyer

“A Decades-Long Journey of Marketing and Public Policy Research to Support the Greater Good” | Maura L. Scott

“Why Don’t Business Schools Publish More Impactful Research?” | Ben McLeish and Mike Taylor

“Some Opportunities for Future Business & Management Research: Employee Health and Well-Being” | Sir Cary L. Cooper

“Efforts to Turn the Tide” | Usha Haley and Andrew Jack

“Medium And Short-term Recommendations to Move Forward” | Usha Haley and Andrew Jack

0 0 votes
Article Rating

Usha Haley and Andrew Jack

Usha Haley is W. Frank Barton Distinguished Chair in International Business, and professor of Management at Wichita State University. She is also director of the Center for International Business Advancement and chair of the World Trade Council of Wichita. She has over 400 publications and presentations on non-market economies, subsidies, multinational corporations, emerging markets, trade, strategy, and scholarly impact, including eight books. Her latest book is Impact and the Management Researcher.
Andrew Jack is a global education editor for the Financial Times, writing on educational issues around the world and editorial lead for the free FT schools program. He was previously head of curated content, deputy editor of the big read section, pharmaceuticals correspondent, and a foreign correspondent in France and Russia.

Notify of

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Inline Feedbacks
View all comments
Would love your thoughts, please comment.x