Criticisms of the Citation System, and Google Scholar in Particular

In the last post I summarised what Google Scholar is and its benefits, in this article I’ll explore some of the common criticisms levelled against the research citation system, and Scholar’s part in that in particular. Of course, some of these issues could be levelled at other services too.

Scholar is bigger and more inclusive than its competitors because it is less curated and more automated, but it has been found to be less accurate. It has also been blamed for not vetting sources and including predatory journals. Search results can be inconsistent and hard to replicate. Some librarians, bibliometricians and university administrators do not recommend using Scholar as the data is deemed “not valid”, and usage guides sometimes come with warnings.

It has been shown it’s possible to spoof Scholar, with fake articles of randomly-generated words accepted, and even a fake author who became the world’s 21st most-highly-cited scientist. Self-citation and routine group citation can also help game metrics, including in Scholar.

Author order is also not currently accounted for in Scholar, a growing issue with many papers now involving high numbers of collaborators. Being a first or sole author is somewhat different to being the 100th, yet Scholar currently gives all three the same weight.

James Wilsdon, of the University of Sheffield, highlighted to me that Scholar, like other citation databases, suffers from an inability to discriminate between different types of citation e.g. this article is completely wrong vs my entire argument is built on this article. Currently both would be treated the same..

Scholar is also criticized for not providing an open application programming interface (API) to other services, and for not sharing its sources, ranking approach or algorithms. The latter practice is still standard in Silicon Valley but completely counter to typical scholarly behavior. In some ways, Scholar is aligned to the Open Access movement, but not quite.

Although Google Scholar metrics can allow comparisons between authors without some biases (e.g. when hiring), it can introduce others – such as between disciplines, where different conventions can result in wildly different metric ‘scores’. You should also be very wary comparing metrics (such as an author’s h-index) between services. Google Scholar and each of its competitors have different coverage, and so give different metric “scores” for authors and journals. Those differences can be patterned in ways relevant to particular disciplines, for instance with Scholar being found to cover fewer pre-1990 publications.

It can be useful for metrics to allow non-experts to evaluate an author’s influence in an academic field, but they can do so without requiring any appreciation of the actual ideas or content of that research area. The inexorable move to quantitative proxies for measuring research success, or financial proxies for real-world impact (consultancy revenues, patents or spin-out companies etc.) can also mask the genuine, often messy, impact of an author’s research on other authors, practitioners or policymakers – but that’s for an article later in the series.

The third and final article on the topic of Google Scholar will look at how the citation system, and Scholar specifically, could be improved.