International Debate

Ethnography’s Denominator Blues International Debate
So this scattering of numbers really represents how good a researcher I am?

Ethnography’s Denominator Blues

August 11, 2021 2999
Quantophrenia

A friendly sociologist recently drew my attention to a scathing review of my book, Interrogating Ethnography. I have realized for the past few years that sociologists tend to be thin-skinned about suggestions from outsiders, while often demonstrating surprisingly poor reading skills. It had not previously occurred to me, however, that someone with a doctorate in social science might lack a fundamental understanding of basic mathematics, such as the role of the denominator. Alas, that seems to have been the case with UCLA’s professor Stefan Timmermans, whose review, titled “Hypocriticism,” appeared in the journal Contemporary Sociology and was cited approvingly in this year’s Annual Review of Sociology.

In what he evidently intended as a devastating riposte, Timmermans wrote:

I tallied up all the fact-checking in this book and grouped them into three categories:

* Lubet doubts ethnographer/finds ethnographer is wrong: 23

* Lubet finds ambiguity/unable to factcheck: 10

* Lubet finds ethnographer is correct/shows ethnographer fact-checking own claims: 22

Considering that this is selectively drawn from 50 ethnographies, each making ‘‘many thousands’’ (p. 136) of claims, Lubet’s indictment of ‘‘ethnographic malpractice’’ (p. 1) falls flat.

Now, it is true that a problem rate of 23 out of thousands would be inconsequential, but it is always possible to trivialize an observation by assuming an exaggerated denominator. What Timmermans did not choose to recognize – although clearly stated multiple times in the book – was that my sample consisted of only those claims that could possibly be fact-checked. Of that number, at least half turned out to be dubious – 23/45 – which is a troubling error rate, to say the least.

Epidemiologists calculate the COVID-19 positivity rate by comparing the number of positive results (numerator) to the number of tests actually administered (denominator). Only a pandemic denier or conspiracy theorist would seek to distort the outcome by expanding the denominator to include the entire population of a locality or state. That sleight of hand might make a disquieting situation appear relatively inconsequential, but it would be misleading and disingenuous.

Ethnography, as I wrote in Interrogating Ethnography, is the only academic discipline – save, perhaps, oral history – that relies on anonymous sources, as reported by a single investigator whose underlying data are never released. Timmermans makes the silly argument that critics of ethnography should be dismissed unless they have done “the real shoe leather work” of reinvestigation. In most cases, of course, that is conveniently impossible, given that ethnographers typically go to great lengths to anonymize their subjects and mask their research sites.

I would be more inclined to attribute Timmermans’s math error to simple negligence, rather than intentional spin, if his other cavils were not so tendentious. He wrote, for example:

Time and time again, he ‘‘fact-checks’’ an ethnographer’s observation with a couple of calls to a prosecutor, police detective, or hospital administrator about a sticky observation. Lo and behold, they deny that the ethnographer ‘‘correctly’’ represented their institution in a possibly negative light. QED: Lubet’s verdict is that the ethnographers were wrong, misled, or naıvely sympathetic with their interlocutors.

Sure, I spoke with the establishment figures Timmermans lists, but I also made far more than “a couple of calls” to public defenders, journalists, housing advocates, economists, educators, and social workers – all unmentioned by Timmermans. Nor does he acknowledge that I also interviewed or corresponded with many of the ethnographers whose work I explored, while cross-checking their observations against census figures, newspaper accounts, court records, school records, hospital records, media records, medical data and CDC reports, historical data, and other available sources. Timmermans’s scare quotes aside, these are exactly the same means used by prominent ethnographers – including Matthew Desmond and Mitchell Duneier – to verify their own work.

Timmermans would have a better point if I had set out to discredit every statement ever made in ethnography, but I intended nothing of the sort. Rather, my objective was to investigate whether ethnography’s characteristic reliance on unverified accounts may sometimes produce misinformation. I reported everything I found, including instances – about half the time – where my doubts were not confirmed. In any other academic discipline, my findings would have provoked less umbrage and more reinvestigation. In ethnography, not so much.

My book was published in late 2017. It would have required very little shoe leather to prove me wrong, if I really was wrong. It would take only a few days for a graduate student to find out whether there are actually police cordons at the doors of Philadelphia hospitals, checking the IDs of young Black men for outstanding warrants (there are not). It would take a little longer to locate a group of Mississippi adolescents, in the 21st century, who are completely unaware of the existence of elevators (because there are none). Social science, like all science, can only be self-correcting if replication is taken seriously. Perhaps Timmermans can explain why that seldom happens in ethnography.

My fact-checking may have been imperfect – although no one has ever identified factual errors – but it can nonetheless help identify potential shortcomings in ethnography or raise valid questions about methodology and conclusions, which is all that I have ever proposed. It is more than a little ironic for Timmermans – a non-physician specializing in medical sociology – to reject suggestions from an outsider who doesn’t “play in our playground.” He needs to spend less time in the sandbox and more time brushing up on math.

Steven Lubet is Williams Memorial Professor at the Northwestern University Pritzker School of Law and author of Interrogating Ethnography: Why Evidence Matters, and other books such as 2015's The “Colored Hero” Of Harper’s Ferry: John Anthony Copeland And The War Against Slavery and Lawyers' Poker: 52 Lessons That Lawyers Can Learn From Card Players. He is the director of the Fred Bartlit Center for Trial Advocacy. He has been living with ME/CFS since 2006.

View all posts by Steven Lubet

Related Articles

From the University to the Edu-Factory: Understanding the Crisis of Higher Education
Industry
November 25, 2024

From the University to the Edu-Factory: Understanding the Crisis of Higher Education

Read Now
Deciphering the Mystery of the Working-Class Voter: A View From Britain
Insights
November 14, 2024

Deciphering the Mystery of the Working-Class Voter: A View From Britain

Read Now
New Initiative Offers Grants for Canadian Research on Research
Announcements
November 5, 2024

New Initiative Offers Grants for Canadian Research on Research

Read Now
Julia Ebner on Violent Extremism
Insights
November 4, 2024

Julia Ebner on Violent Extremism

Read Now
Emerson College Pollsters Explain How Pollsters Do What They Do

Emerson College Pollsters Explain How Pollsters Do What They Do

As the U.S. presidential election approaches, news reports and social media feeds are increasingly filled with data from public opinion polls. How […]

Read Now
Diving Into OSTP’s ‘Blueprint’ for Using Social and Behavioral Science in Policy

Diving Into OSTP’s ‘Blueprint’ for Using Social and Behavioral Science in Policy

Just in time for this past summer’s reading list, in May 2024 the White House Office of Science and Technology Policy (technically, […]

Read Now
All Change! 2024 – A Year of Elections: Campaign for Social Science Annual Sage Lecture

All Change! 2024 – A Year of Elections: Campaign for Social Science Annual Sage Lecture

With over 50 countries around the world holding major elections during 2024 it has been a hugely significant year for democracy as […]

Read Now
2.3 3 votes
Article Rating
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Newest
Oldest Most Voted
Inline Feedbacks
View all comments