Academic Funding

The Nonresponse Challenge to Surveys and Statistics

May 7, 2013 3290

Special Editors:  Douglas S. Massey and Roger Tourangeau

Informed social policymaking relies on survey-based research. Understanding the effects of mass immigration, for example, or responding to the changing nature of the American workforce, or addressing the health and educational needs of contemporary American families requires good data that tell us about American society.

Most publicly reported statistics come from survey samples, not from tabulations of administrative data or actual population counts.  For example, almost all of what people commonly refer to as “census data” does not come from an actual enumeration of the U.S. population but from a sample survey that is administered along with the census—the 2010 U.S. Census contained just four questions about the household and seven questions about each household member—all other data now come from the American Community Survey (ACS), which samples about 2 million households. In addition to the census, other federal sample surveys include the Current Population Survey (which produces monthly labor force statistics), the National Survey of Family Growth (birth and childbearing statistics), and the National Health Interview Survey (health statistics). The federal government also supports other surveys indirectly through grants, including the New Immigrant Survey, the Panel Study of Income Dynamics, and the American National Election Study.

But survey researchers are increasingly unable to get people to respond to surveys.  This is a real worry because nonresponse can lead to biased research and because nonresponse poses a significant threat to the federal statistical system in its entirety.

This volume of The ANNALS abundantly highlights the critical importance of surveys to contemporary society, addresses the causes and consequences of rising survey nonresponse, and explores ways to address the challenges that nonresponse brings.

Key Observations and Findings

  • High rates of nonresponse do not necessarily translate into bias.  Research in this volume shows how response rates by themselves are poor indicators of bias (in some cases, raising response rates can even increase bias).  But rising nonresponse increases the likelihood of bias, and survey nonresponse is increasing at an alarming rate.
  • The survey nonresponse problem is likely going to continue to worsen, and survey costs are likely going to continue to rise. We show that nonresponse is related with the frequency of single-person households, the prevalence of households without children, and long commute times—all seemingly endemic to social life today. The increase in nonresponse is driven by powerful generational changes: younger cohorts are associated with low response and less comfortable with communication techniques favored by survey researchers, such as telephone surveys and face-to-face interviews.
  • The simplest way to address nonresponse is to reward respondents financially for their participation. Evidence clearly indicates that financial incentives increase response rates, but this also increases costs.
  • Reliance on nonsurvey data can assist in getting quality survey-based estimates.  Using administrative records and “paradata” to strengthen and extend survey research are critical ways to adjust survey estimates and reduce budget pressures on the survey organizations. It may also reduce the level of survey fatigue in the general public. Interestingly, the most important limitation on the potential use of administrative systems as data for nonresponse adjustment is legal: federal legislation prohibits the public release of the data collected by the federal government, so it can only be done in-house by the Census Bureau or some other authorized agency.
  • Rising costs and likely federal budget cuts will probably mean serious efforts to overhaul one or more major federal surveys (instruments that have not undergone major redesigns in decades) or to consolidate federal data collection into fewer surveys (or both). The current model for many high-quality but costly surveys may soon become unsustainable. Research in this volume shows that response rates for the most difficult and expensive kinds of surveys—longitudinal studies that follow the same respondents and reinterview them over time—have not seen the same increases in nonresponse rates as have been observed for one-time surveys.
  • The volume is replete with technical details of how to maximize the likelihood of response and compensate for nonresponse. One promising analytic technique highlighted in the volume is the multi-level integrated database approach, which employs multiple databases to collect as much information as possible about a target population to maximize the accuracy of survey estimates. Data systems integration is definitely a direction in which the field is moving.

Perhaps the greatest threat to the future of survey data is the lack of public recognition of the importance of statistical and scientific surveys in the world today.  It is probably time for an industry-wide effort to improve the image of survey research and, critically, to differentiate legitimate social scientific surveys from the onslaught of unwanted solicitations that most Americans are trying so hard to fend off.

Protecting the integrity of survey research also requires better efforts to educate our political leadership and society writ large. Must all become better educated to the critical importance of social surveys in contemporary society, not only because they provide basic information to inform citizen-voters in a democratic society but also because they generate key knowledge for American businesses that enable us to compete effectively in a global, knowledge-based economy.  We offer this volume as a first step in a broader and more concerted effort to educate policy-makers and the public about the key importance of social surveys to the healthy functioning of postindustrial society.

Issue Conclusion:

Where Do We Go from Here? Nonresponse and Social Measurement

Written Statement of Paul Emrath:

On Behalf of the National Association of Home Builders, Impact of Falling Response Rates to Social Surveys and What Can Be Done

The American Academy of Political and Social Science, one of the nation’s oldest learned societies, is dedicated to the use of social science to address important social problems. For over a century, our flagship journal, The Annals of the American Academy of Political and Social Science, has brought together public officials and scholars from across the disciplines to tackle issues ranging from racial inequality and intractable poverty to the threat of nuclear terrorism. Today, through conferences and symposia, podcast interviews with leading social scientists, and the annual induction of Academy Fellows and presentation of the Daniel Patrick Moynihan Prize, the Academy is dedicated to bridging the gap between academic research and the formation of public policy.

View all posts by American Academy of Political and Social Science

Related Articles

Why Don’t Algorithms Agree With Each Other?
Innovation
February 21, 2024

Why Don’t Algorithms Agree With Each Other?

Read Now
A Black History Addendum to the American Music Industry
Insights
February 6, 2024

A Black History Addendum to the American Music Industry

Read Now
SSRC Links with U.S. Treasury on Evaluation Projects
Announcements
February 1, 2024

SSRC Links with U.S. Treasury on Evaluation Projects

Read Now
The Use of Bad Data Reveals a Need for Retraction in Governmental Data Bases
Communication
February 1, 2024

The Use of Bad Data Reveals a Need for Retraction in Governmental Data Bases

Read Now
Connecting Legislators and Researchers, Leads to Policies Based on Scientific Evidence

Connecting Legislators and Researchers, Leads to Policies Based on Scientific Evidence

The author’s team is developing ways to connect policymakers with university-based researchers – and studying what happens when these academics become the trusted sources, rather than those with special interests who stand to gain financially from various initiatives.

Read Now
NSF Responsible Tech Initiative Looking at AI, Biotech and Climate

NSF Responsible Tech Initiative Looking at AI, Biotech and Climate

The U.S. National Science Foundation’s new Responsible Design, Development, and Deployment of Technologies (ReDDDoT) program supports research, implementation, and educational projects for multidisciplinary, multi-sector teams

Read Now
There’s Something In the Air…But Is It a Virus? Part 1

There’s Something In the Air…But Is It a Virus? Part 1

The historic Hippocrates has become an iconic figure in the creation myths of medicine. What can the body of thought attributed to him tell us about modern responses to COVID?

Read Now
0 0 votes
Article Rating
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

1 Comment
Newest
Oldest Most Voted
Inline Feedbacks
View all comments