Business and Management INK

What Does Trustworthy and Credible MTurk Research Look Like? Recommendations and Checklist

November 30, 2020 3805
Drawing of the original mechanical Turk
Unveiled in the late 1700s, the original Mechanical Turk, while ultimately shown to be a hoax, was an automaton that played chess.

Did you know that the use of MTurk for research purposes has increased more than 20 times in the past 10 years? Are you interested in using MTurk for your own research and want to make sure you do things right? Have you recently read a study that used an MTurk sample and are wondering whether you can trust its results and conclusions? Are you a journal reviewer who needs to provide constructive feedback to authors on how to improve the quality of their MTurk-based research and would like to know how to go about doing so?

Our Journal of Management article, “MTurk Research: Review and Recommendations,” provides answers to each of these questions. We do so based on two separate literature reviews involving more than 600 sources. Our review includes articles and books in management, accounting, consumer behavior, medical research, organizational psychology, political science, sociology and other fields.

This video abstract from Herman Aguinis, Isabel Villamor and Ravi S. Ramani details their paper just published online in the Journal of Management.

What did we find? First, it’s clear that MTurk’s incredible popularity is based on four main benefits. First, you can use a large and diverse participant pool. For example, MTurk allows researchers access to a larger and more demographically diverse participant pool. For example, compared to student samples, MTurkers are older, have more years of relevant work experience, and report greater computer and internet knowledge. Second, you can access and collect data very quickly. For example, most MTurk assignments are completed within 12 hours or less because, at any given time, there are about 7,300 MTurkers available. Third, MTurk is quite inexpensive compared to other data collection methods. For example, recruiting 100 participants for a survey that takes 10 minutes to complete costs around $120 plus a 20 percent fee for MTurk—a total of only $145. Finally, MTurk gives you flexibility in terms of research design choices. For example, MTurk can be used to implement experimental, passive observation, quasi-experimental, and longitudinal research designs.

But, despite these fabulous benefits, our reviews uncovered 10 critical validity-related challenges that researchers using MTurk face and must address for conclusions to be credible: (1) inattention, (2) self-misrepresentation, (3) self-selection bias, (4) high attrition, (5) inconsistent English language fluency, (6) non-naiveté, (7) growth of MTurker communities, (8) vulnerability to web robots (or “bots”), (9) social desirability bias, and (10) perceived researcher unfairness.

Herman Aguinis, left, Isabel Villamor and Ravi S. Ramani

Our article offers evidence-based recommendations for addressing each of the 10 challenges. We took great care in creating this checklist with recommendations that are practical and actionable—not just “pie in the sky.” For example, to address the threats of inattention and vulnerability to web robots, we suggest including a CAPTCHA verification question—a question where MTurkers provide a self-report of response effort, as well as a question requiring an open-ended response. As a second example, to gauge social desirability, we recommend repeating pertinent questions at the end of the study that make explicit the desired response, and contrasting participant answers to the same questions as when presented earlier. As yet another example of solutions, to counter the threat of careless or insufficient effort responding, we recommend analyzing response times and excluding participants who completed the task in less than one or two seconds per item. We offer similarly detailed and actionable solutions for dealing with each of the 10 critical challenges.

Our article is useful for those who want to use MTurk for the first time, as well as for those who have already used MTurk but wish to improve the quality of their future research. Also, journal editors and reviewers can use the checklist to evaluate a submission and provide developmental feedback to increase transparency and reproducibility. In addition, consumers of MTurk research, including other researchers but also practitioners and policy makers, can use our checklist to understand the extent to which a study has followed best practices and therefore can be trusted.

Overall, we hope our Journal of Management article will serve as a catalyst for more robust, reproducible, and trustworthy MTurk-based research in management and other fields. Please do get in touch with us if you have any questions!

Herman Aguinis (pictured) is the Avram Tucker Distinguished Scholar, professor of management, and chairperson of the Department of Management at The George Washington University School of Business. He has been elected for the presidency track of the Academy of Management. Isabel Villamor is a Ph.D. candidate in the Department of Management at GWU. Ravi S. Ramani is an assistant professor of organizational behavior and human resource management in the Business Administration department of the Earl G. Graves School of Business and Management at Morgan State University.

View all posts by Herman Aguinis, Isabel Villamor, and Ravi S. Ramani

Related Articles

‘Push, Pull, Dance’: Public Health Procurement – Saving Lives and Preventing Harm
Business and Management INK
June 12, 2024

‘Push, Pull, Dance’: Public Health Procurement – Saving Lives and Preventing Harm

Read Now
Beyond Net-Zero Targets: When Do Companies Maximize Their Potential to Reduce Carbon Emissions?
Business and Management INK
June 4, 2024

Beyond Net-Zero Targets: When Do Companies Maximize Their Potential to Reduce Carbon Emissions?

Read Now
How AI-Integration is Changing the Workplace
Business and Management INK
May 28, 2024

How AI-Integration is Changing the Workplace

Read Now
Keeping Qualitative Research Weird!
Business and Management INK
May 23, 2024

Keeping Qualitative Research Weird!

Read Now
Sometimes, We Do Need a Narcissist

Sometimes, We Do Need a Narcissist

Karynne Turner, Feray Adigüzel, and Jatinder S Sidhu reflect on their research article, “Chief executive officer narcissism, corporate inertia, and securities analysts’ stock […]

Read Now
From Collision to Collaboration: Bridging University and Industry Relationships

From Collision to Collaboration: Bridging University and Industry Relationships

In this article, Will Harvey and Paul Spee reflect on the importance of collaboration between industry and universities. This topic was the catalyst for their research article, “Walking the tightrope of academic and practitioner expectations in field research,” found in Management Learning.

Read Now
Motivation of Young Project Professionals: Their Needs for Autonomy, Competence, Relatedness, and Purpose

Motivation of Young Project Professionals: Their Needs for Autonomy, Competence, Relatedness, and Purpose

Young professionals born between the early 1980s and mid-1990s now constitute a majority of the project management workforce. Having grown up connected, collaborative, and mobile, they have specific motivations and needs, which are explored in this study.

Read Now
4.7 3 votes
Article Rating
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Inline Feedbacks
View all comments