Lumivero Report: Qualitative Researchers Adopting AI, But Lack Confidence In It
“Trust but verify” is an old Russian saying that U.S. President Ronald Reagan made famous when discussing treaties to limit nuclear weapons. That idea reflects the feelings of researchers, faculty, students and practitioners in a recent survey about artificial intelligence: a majority are using AI in some fashion, but they’re using it grudgingly and keeping it within eyeshot. And they’re supplying the labor required to verify the AI results.
As a section header late in a report on the survey reads, the “bottom line” is “adoption without confidence.”
The survey and resulting report, “The State of AI in Qualitative and Mixed Methods Research,” was conducted earlier this year by Lumivero, a maker of research software such as the well-known qualitative package NVivo. The company has recently released its own AI-native package, NVivo AI Cloud, shortly before the report’s own release.
Lumivero surveyed 115 researchers in both academe and private industry. More than half those surveyed said they used AI regularly (21%) or occasionally (36%), and another 19% reported trying out AI a time or two.
Had you thought new technology might appeal to younger researchers, think again. The most regular users, according to these results, were senior researchers, and the least likely to adopt were the most junior. Specifically, one third of researchers with 11–20 years in the field were regular users as were 30% of those with more than 20 years’ experience. Among those with less than 2 years’ experience, however, 44% percent had not tried AI.
Among the respondents’ AI use cases, for doctoral researchers it was as a writing assistant, in the government and public sector it was research support and transcription, and among private industry it was coding and thematic analysis. (While the pool of respondents wasn’t large enough for disciplinary breakdowns, the report did note that healthcare was particularly resistant to AI, with 58% having not yet used AI or choosing not to use it.
The use of AI in qualitative research is not without its critics. Last November, an open letter titled “We Reject the Use of Generative Artificial Intelligence for Reflexive Qualitative Research” appeared in the journal Qualitative Inquiry. Well-known qualitative researchers Tanisha Jowsey , Virginia Braun, Victoria Clarke, Deborah Lupton, and Michelle Fine, in an open letter signed by 419 qualitative researchers from 32 countries, argued that AI lacks real understanding and cannot replace the inherently subjective, meaning-driven nature of human analysis. Furthermore, they oppose GenAI on social and environmental justice grounds, citing its severe ecological impacts and the exploitation of labor in the Global South. The letter generated a lively response, in and out of the journal.
In the Lumivero survey, the concerns about AI were focused on its outputs and integrity. “Across all respondents, 87% are concerned about accuracy and hallucination risk, 86% about research integrity, and 85% about data privacy—concerns that intensify, rather than ease, at the analytical core of the work,” the report’s authors wrote. “Trust is highest where AI’s output is easiest to verify, and lowest wherever a task requires interpretation.” Speaking of verification, some 61% of respondents said they manually review AI output against source data, and 59% cross-check it against their own independent analysis.
How AI gets used also seems to hinge on experience. Use tended to concentrate in areas where individual researchers already knew the terrain reasonably well and presumably could best assess AI outputs. “Researchers want transparency,” explained sociologist Silvana di Gregorio, Lumivero’s product research director and head of qualitative research, in an interview. “They want the AI to be able to show them their work. They don’t want a black box.”
And so, the most common use of AI among regular and occasional users is literature review (71%), followed by transcription (58%). The least common use is interpretation and sense-making. Returning to the idea of trust but verify, the highest marks for high or very high trust in AI’s ability accrued to summarizing coded material, transcription or data preparation, while the lowest marks in trust went to interpretation and sense-making.
However, while those high marks (34%) were “the one task in the survey where trust in AI’s output outweighs distrust,” the authors write, adding that this “mak(es) it the closest thing to a settled, comfortable use case researchers described,” it wasn’t all that much greater than the low-water mark of 20% for interpretation.
The survey clearly identified the wellsprings of that distrust. “Across all respondents,” the survey reads, “87% are concerned about accuracy and hallucination risk, 86% about research integrity, and 85% about data privacy—concerns that intensify, rather than ease, at the analytical core of the work. Trust is highest where AI’s output is easiest to verify, and lowest wherever a task requires interpretation.”
And distrust breeds disuse. “Distrust of outputs, not cost or training gaps,” the report reads, “is the most-cited reason researchers don’t use AI more—a finding that holds in the general sample and, at a lower intensity, in government, healthcare, and commercial settings, too.” Some 57% of respondents identified distrust of AI outputs as their greatest obstacle to adoption. Privacy and data security concerns were next at 51%, followed by cost and then lack of training.
“Institutional policy exists more often than confidence in AI does,” according to the report. “70% of researchers report their organization has some form of AI policy—38% specifically permit use with disclosure or review requirements—but policy clarity hasn’t fully translated into consistent adoption. Among researchers who’ve tried AI without adopting it consistently, the largest share—45%—work at institutions with restrictive AI policies, more than double any other policy type.”
The report urges observers to “’look closely at what researchers are actually doing…
“They verify AI’s output against source material. They translate its suggestions into their own analytical language before adopting them. They bound its role—deciding what AI may touch and what remains theirs alone. And they pressure-test its conclusions against their own independent reading of the data. These are not workarounds or reluctant extra labor.”
While that suggests some thought has been put into “dealing with” AI, but it’s not showing up consistently along the research food chain. For example, di Gregorio isn’t seeing consistency or clarity from the institutional review boards that approve research projects.
“It’s totally inconsistent,” she said. “I did a bit of research this year trying to find out about that and there’s no consensus basically.” For example, while research subjects may be asked if they will consent to being recorded, generally they’re not asked if they can be analyzed by AI. “Do you put that in the consent form? I haven’t seen that. It’s because [researchers] followed their institutions regulations for consent. But it hasn’t been updated. So there’s a whole area here that hasn’t been figured out yet.”
Di Gregorio has been doing her part to fill in the gaps, “researching on AI and speaking at conferences and that sort of thing.” For the last six months she’s been working on the tool, NVivo AI Cloud, mentioned above, which she says addresses many of the concerns, such as transparency, that haunt researchers. And she suspects AI will eventually find its place in the research toolkit.
“I think it’s going to change methodology, and it will develop new methodologies. The analogy I always make is when the portable tape recorder came out in the 1960s. That enabled for the first time that you could actually analyze talk — and the whole the whole field of conversational analysis developed; that technological development enabled that form of analysis. And I think I think it’s the same thing here: new forms of methodology in social science, certainly in qualitative social science research, will start to evolve.”
And for those wondering … Luminvo explained that for the project, “the research and analysis were human-led, AI-assisted. Lumivero’s in-house experts wrote the survey questions, and the survey was hosted on SurveyToGo. It was an online survey that included open-ended questions and was not supplemented by interviews. “
The Lumivero team conducted the analysis using NVivo and XLSTAT. The report was then edited and double-checked by a human, with AI assisting. NVivo AI Cloud was not yet available for the team to use for this analysis over the summer.
