Career

Proof Over Promise: Finding An Impact Factor for Your Career Career
If only there were some cheap, easy-to-find and easy-to-use tool out there ...

Proof Over Promise: Finding An Impact Factor for Your Career

September 18, 2014 2515

Google Scholar logo

If only there were some cheap, easy-to-find and easy-to-use database out there …

Economists love to rank. Even a casual five-minute literature search reveals literally hundreds of publications on rankings of academic productivity and impact. Dutch economists are no exception. In fact, they produced what, to our best knowledge, is the oldest ranking in the field: a nation-wide ranking of economists (the Economentop 40, or Economics Top 40) that has entered its fourth decade. However, this ranking is based on publication volume in ISI-listed journals weighted by the journal’s impact factor or article influence score, rather than on the actual impact of the publications in question.

In this blog we explain the added value of a career-impact focus and envision some practical implications for business schools. We will show that an impact driven approach using Google Scholar creates a more democratic and inclusive assessment.

Proof (impact) and Promise (journals)

On average, publications in high-impact journals by definition get cited more frequently than publications in low-impact journals as the journal impact factor is based on average citations. This still is the fundamental logic behind the creation and maintenance of journals list by business schools and associations such as the Association of Business Schools. We call this principle “promise”, i.e. publishing in a high-impact journal carries the implicit promise that the article will also be highly cited. However, not all individual papers published in these high-impact journals will fulfill this promise. In our paper “Proof over promise: Towards a more inclusive ranking of Dutch academics in Economics & Business,” we therefore set out to create a ranking based on “proof,” i.e. rather than looking at the promised number of citations implied by the journal impact factor or article influence score, we look at actual citations to an author’s work.

LSE Impact logo

This article by Anne-Wil Harzing and Wilfred Mijnhardt originally appeared on the LSE Impact of Social Sciences blog as “Proof over promise: Moving citation metric systems beyond journal impact towards a career impact approach” and is reposted under the Creative Commons license (CC BY 3.0).

Introducing two new impact metrics with individualized career focus

In our paper we use two relatively new metrics, the citations per author per year (CAY) metric and the individual annual h-index (hIa, see Harzing, Alakangas & Adams, 2014 for details). These two metrics are the individualized and annualized equivalents of respectively total citations and the traditional h-index, two important bibliometric measures. We correct our metrics for the number of co-authors as the number of co-authors positively influences both the number of publications an academic is able to publish and the number of citations. A correction for the number of years an academic has been active is important as citations continue to increase over an academic career.

hIa career impact: field independent focus on contribution

In this blog, we focus on the hIa index, which is calculated by dividing the individual h-index (an h-index corrected for the number of co-authors) by the number of years an academic has been active, i.e. the number of years that have lapsed since their first publication. The metric thus represents the average number of single-author-equivalent “impactful” articles that an academic has published per year and hence permits an intuitive interpretation. Based on an empirical example of 146 academics in five major disciplines at different career stages, Harzing, Alakangas & Adams (2014) showed that the hIa index attenuates h-index differences that are purely attributable to (disciplinary) co-authorship practices and career lengths. As such, this metric provides a more reliable comparison between academics in different disciplines and at different career stages than the h-index.


Further details

Harzing, A.W.; Mijnhardt, W. Proof over promise: Towards a more inclusive ranking of Dutch academics in Economics & Business, in press for Scientometrics, DOI: 10.1007/s11192-014-1370-z, prepublication version available at http://www.harzing.com/papers.htm#top40


A second aspect of our focus on proof over promise is the use of Google Scholar rather than ISI as a data source. Although ISI listing is seen by many to imply a quality stamp, in our view it should not matter where research is published. If a particular research output is highly cited, it clearly influences the field and that should be more important than the journal in which it is published. As Google Scholar on its own is not very suitable for bibliometric analyses, Publish or Perish (Harzing 2007) was used to collect citation data from Google Scholar. There are now more than 500 published articles referring to the Publish or Perish program. This indicates that—in spite of its limitations—Google Scholar is perceived to be a useful source of bibliometric data.

Resulting ranking for economics & business in the Netherlands

The resulting hIa-based ranking of academics in economics & business in the Netherlands (see Harzing & Mijnhardt, in press for details) is substantially different from the original ranking based on publications and is more inclusive in terms of disciplines, age and affiliation.

  • Whereas in the original publication-based Economics top-40 more than two thirds of the listed academics works in Economics or Management Science, this proportion is reduced to 43 percent in the hIa ranking. The change in disciplinary composition is particularly striking in the top 20. In the publication-based Economics Top 40, three quarters of the academics in the top 20 were economists, in the PoP hIa top 40 this is reduced to just over a third.
  • At 47, the average age in the hIa top 40 is lower than in the original ranking (50), but most noticeable is the addition an additional five academics under 40 and another five aged between 40 and 45. In fact, all but three of the newly listed academics in this top 40 are aged 45 or under.
  • In the original top 40 academics affiliated with Erasmus and Tilburg made up just over or just under half of the list, in the hIa top 40, this is reduced to a third. Apart from the University of Utrecht, every university is represented in the hIa top 40. Maastricht, Eindhoven and the UvA (University of Amsterdam) in particular do much better in citation-based rankings than in the original publication-based top 40.

We argue that our “proof over promise” approach is more “democratic”/inclusive than the original Economics Top 40. First, by expanding the type of research outputs considered beyond the narrow scope of publications in ISI-listed journals, we remove the disciplinary bias against management, marketing and accounting & finance, disciplines in which a smaller proportion of high-quality journals is ISI-listed than in economics and management science (Harzing & van der Wal, 2009).

Second, citation-based performance metrics can be argued to be more democratic as their “verdict” is based on the reception of the paper by the academic community as a whole, whereas acceptance in a high-impact journal is dependent on only a handful of gatekeepers (the editor and reviewers).

Third, our ranking was conducted with a free software program (Publish or Perish) and a publicly available database (Google Scholar). Hence, any reader can easily replicate the ranking without the need for subscription-based databases. This also means that any academic can look up their own citation record and easily find out where they score in the current ranking

Fourth, citation-based rankings and in particular the hIa ranking are likely to provide more dynamic rankings in terms of changes over the years. Younger academics can more easily enter into the hIa ranking if they perform well relative to their career length in terms of single-author equivalent impactful papers.

As for other research metrics, the CAY or hIa-index should never be used as the sole criterion to evaluate academics. Another crucial question that should always be asked is: “Has the scholar asked an important question and investigated it in such a way that it has the potential to advance societal understanding and well-being?” (see e.g. Adler and Harzing, 2009). However, we argue that the hIa-index provides an important additional perspective over and above a ranking based purely on publications in high impact journals alone.

Implications for business schools

Using metrics such as the CAY and hIa, business schools can move away from the journal-based logic and start thinking from an impact -driven logic. This could have important consequences for incentive policies, mainly for senior faculty. We advice deans of business schools to start assessing their senior faculty from an impact perspective, instead of the narrow journal-based assessment. This approach is not only more inclusive, it is also more democratic and gives more opportunity for diversity to flourish, especially in business schools with a diverse social sciences background.


Anne-Wil Harzing is research professor and research development advisor at ESCP Europe and professor of international management at Middlesex University. Her research interests include international HRM, HQ-subsidiary relationships, the role of language in international business, and the quality and impact of academic research. She maintains an extensive website (www.harzing.com) with resources for international management and academic publishing. Wilfred Mijnhardt is policy director at Rotterdam School of Management and executive director of the Erasmus Research Institute of Management at Erasmus University Rotterdam. He publishes a weblog Research Management in Management Research and shares his ideas on social media @wmijnhardt

View all posts by Anne-Wil Harzing and Wilfred Mijnhardt

Related Articles

Three Decades of Rural Health Research and a Bumper Crop of Insights from South Africa
Impact
March 27, 2024

Three Decades of Rural Health Research and a Bumper Crop of Insights from South Africa

Read Now
Using Translational Research as a Model for Long-Term Impact
Impact
March 21, 2024

Using Translational Research as a Model for Long-Term Impact

Read Now
2024 Holberg Prize Goes to Political Theorist Achille Mbembe
News
March 14, 2024

2024 Holberg Prize Goes to Political Theorist Achille Mbembe

Read Now
Edward Webster, 1942-2024: South Africa’s Pioneering Industrial Sociologist
Career
March 12, 2024

Edward Webster, 1942-2024: South Africa’s Pioneering Industrial Sociologist

Read Now
Charles V. Hamilton, 1929-2023: The Philosopher Behind ‘Black Power’

Charles V. Hamilton, 1929-2023: The Philosopher Behind ‘Black Power’

Political scientist Charles V. Hamilton, the tokenizer of the term ‘institutional racism,’ an apostle of the Black Power movement, and at times deemed both too radical and too deferential in how to fight for racial equity, died on November 18, 2023. He was 94.

Read Now
Norman B. Anderson, 1955-2024: Pioneering Psychologist and First Director of OBSSR

Norman B. Anderson, 1955-2024: Pioneering Psychologist and First Director of OBSSR

Norman B. Anderson, a clinical psychologist whose work as both a researcher and an administrator saw him serve as the inaugural director of the U.S. National Institute of Health’s Office of Behavioral and Social Sciences Research and as chief executive officer of the American Psychological Association, died on March 1.

Read Now
New Feminist Newsletter The Evidence Makes Research on Gender Inequality Widely Accessible

New Feminist Newsletter The Evidence Makes Research on Gender Inequality Widely Accessible

Gloria Media, with support from Sage, has launched The Evidence, a feminist newsletter that covers what you need to know about gender […]

Read Now
0 0 votes
Article Rating
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Inline Feedbacks
View all comments