Higher Education Reform

Seeking a Research Rating That Isn’t Impact Factor

June 13, 2014 1470

Dora the Explorer

DORA, in this case the people behind the San Francisco Declaration on Research Assessment, is exploring ways to replace the impact factor as a metric for judging science. (Håkan Dahlström) / CC BY 2.0

This article was contributed by the authors of the San Francisco Declaration on Research Assessment: David Drubin (University of California, Berkeley; Molecular Biology of the Cell), Stefano Bertuzzi (American Society for Cell Biology), Michael Marks (Children’s Hospital of Philadelphia; Traffic), Tom Misteli (National Cancer Institute; The Journal of Cell Biology), Mark Patterson (eLife), Bernd Pulverer (EMBO Press), Sandra Schmid (University of Texas Southwestern Medical Center).

***

Scientists, like other professionals, need ways to evaluate themselves and their colleagues. These evaluations are necessary for better everyday management: hiring, promotions, awarding grants and so on. One evaluation metric has dominated these decisions, and that is doing more harm than good.

This metric, called the journal impact factor or just impact factor, and released annually, counts the average number of times a particular journal’s articles are cited by other scientists in subsequent publications over a certain period of time. The upshot is that it creates a hierarchy among journals, and scientists vie to get their research published in a journal with a higher impact factor, in the hope of advancing their careers.

The Conversation logo_AU

This article by David Drubin originally appeared at The Conversation, a Social Science Space partner site, under the title “Time to discard the metric that decides how science is rated”

The trouble is that impact factor of journals where researchers publish their work is a poor surrogate to measure an individual researcher’s accomplishments. Because the range of citations to articles in a journal is so large, the impact factor of a journal is not really a good predictor of the number of citations to any individual article. The flaws in this metric have been acknowledged widely – it lacks transparency and, most of all, it has unintended effects on how science gets done.

A recent study that attempted to quantify the extent to which publication in high-impact-factor journals correlates with academic career progression highlights just how embedded the impact factor is. While other variables also correlate with the likelihood of getting to the top of the academic ladder, the study shows that impact factors and academic pedigree are rewarded over and above the quality of publications. The study also finds evidence of gender bias against women in career progression and emphasizes the urgent need for reform in research assessment.

Judging scientists by their ability to publish in the journals with the highest impact factors means that scientists waste valuable time and are encouraged to hype up their work, or worse, only in an effort to secure a space in these prized journals. They also get no credit for sharing data, software and resources, which are vital to progress in science.

This is why, since its release a year ago, more than 10,000 individuals across the scholarly community have signed the San Francisco Declaration on Research Assessment (DORA), which aims to free science from the obsession with the impact factor. The hope is to promote the use of alternative and better methods of research assessment, which will benefit not just the scientific community but society as a whole.

The DORA signatories originate from across the world, and represent just about all constituencies that have a stake in science’s complex ecosystem – including funders, research institutions, publishers, policymakers, professional organizations, technologists and, of course, individual researchers. DORA is an attempt to turn these expressions of criticism into real reform of research assessment, so that hiring, promotion and funding decisions are conducted rigorously and based on scientific judgements.

We can also take heart from real progress in several areas. One of the most influential organizations that is making positive steps towards improved assessment practices is the US National Institutes of Health. The specific changes that have come into play at the NIH concern the format of the CV or “biosketch” in grant applications. To discourage the grant reviewers focusing on the journal in which previous research was published, NIH decided to help reviewers by inserting a short section into the biosketch where the applicant concisely describes their most significant scientific accomplishments.

At the other end of the spectrum, it is just as important to find individuals who are adopting new tools and approaches in how they show their own contributions to science. One such example is Steven Pettifer, a computer scientist at University of Manchester, who gathers metrics and indicators, combining citations in scholarly journals with coverage in social media about his individual articles to provide a richer picture of the reach and influence of his work.

Another example, as reported in the journal Science, comes from one of the DORA authors, Sandra Schmid at the University of Texas Southwestern Medical Center. She conducted a search for new faculty positions in the department that she leads by asking applicants to submit responses to a set of questions about their key contributions at the different stages in their career, rather than submitting a traditional CV with a list of publications. A similar approach was also taken for the selection of the recipients for a prestigious prize recognizing graduate student research, the Kaluza Prize.

These examples highlight that reform of research assessment is possible right now by anyone or any organization with a stake in the progress of science.

One common feature among funding agencies with newer approaches to research assessment is that applicants are often asked to restrict the evidence that supports their application to a limited number of research contributions. This emphasizes quality over quantity. With fewer research papers to consider, there is greater chance that the evaluators can focus on the science, rather than the journal in which it is published. It would be encouraging if more of these policies also explicitly considered outputs beyond publications and included resources such as major datasets, resources and software, a move made by the US National Science Foundation in January 2013. After all, the accomplishments of scientists cannot be measured in journal articles alone.

There have been at least two initiatives that focus on metrics and indicators at the article level, from US standards’ agency NISO and UK’s higher education body HEFCE. Although moves towards a major reliance on such metrics and indicators in research assessment are premature, and the notion of an “article impact factor” is fraught with difficulty, with the development of standards, transparency and improved understanding of these metrics, they will become valuable sources of evidence of the reach of individual research outputs, as well as tools to support new ways to navigate the literature.

As more and more examples appear of practices that don’t rely on impact factors and journal names, scientists will realize that they might not be as trapped by a single metric as they think. Reform will help researchers by enabling them to focus on their research and help society by improving the return on the public investment in science. The Conversation


David Drubin is a professor of cell and developmental biology at the University of California, Berkeley. The Drubin and Barnes Lab is interested in the molecular mechanisms that underlie highly dynamic actin-mediated membrane trafficking events. These studies are being carried out in mammalian cells and in budding yeast. The approaches employed for these studies include state-of-the-art real-time image analysis of live cells, genome-wide functional analyses, genetics, molecular genetics, and biochemistry.

View all posts by David G. Drubin

Related Articles

Daniel Kahneman, 1934-2024: The Grandfather of Behavioral Economics
News
March 27, 2024

Daniel Kahneman, 1934-2024: The Grandfather of Behavioral Economics

Read Now
Using Translational Research as a Model for Long-Term Impact
Impact
March 21, 2024

Using Translational Research as a Model for Long-Term Impact

Read Now
2024 Holberg Prize Goes to Political Theorist Achille Mbembe
News
March 14, 2024

2024 Holberg Prize Goes to Political Theorist Achille Mbembe

Read Now
New Feminist Newsletter The Evidence Makes Research on Gender Inequality Widely Accessible
Impact
March 4, 2024

New Feminist Newsletter The Evidence Makes Research on Gender Inequality Widely Accessible

Read Now
Why Don’t Algorithms Agree With Each Other?

Why Don’t Algorithms Agree With Each Other?

David Canter reviews his experience of filling in automated forms online for the same thing but getting very different answers, revealing the value systems built into these supposedly neutral processes.

Read Now
Contemporary Politics Focus of March Webinar Series

Contemporary Politics Focus of March Webinar Series

This March, the Sage Politics team launches its first Politics Webinar Week. These webinars are free to access and will be delivered by contemporary politics experts —drawn from Sage’s team of authors and editors— who range from practitioners to instructors.

Read Now
Addressing the United Kingdom’s Lack of Black Scholars

Addressing the United Kingdom’s Lack of Black Scholars

In the UK, out of 164 university vice-chancellors, only two are Black. Professor David Mba was recently appointed as the first Black vice-chancellor […]

Read Now
0 0 votes
Article Rating
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Inline Feedbacks
View all comments