Gathering Data for Policy Makers, Business and the Public

Federal surveys have been getting more expensive to administer, in part because the number of people who actually respond to surveys has been progressively declining.  As a result, researchers are developing and implementing new methods for collecting the information that our policy makers, business leaders and the public officials depend on. Many of these new approaches are highlighted in volume 645 (January 2013) of the Annals of the American Academy of Political and Social Science.  These approaches use new technologies and combine strategies to collect needed data. Below is an explanation of some of the technical terms and new approaches presented in the Annals.

Multi-modal Surveys

Combining several different methods of approaching people for surveys is a promising approach to maximize response rates. For example, the American Community Survey now is conducted through a multi-modal survey. First, a statistically valid random sample of people to be surveyed is selected from an address database. These people are mailed a letter asking them to fill out the survey on a secure website. Response rates for the website are fairly high, especially considering that the cost of this first step is very low. If people don’t respond, they are mailed a survey they can submit by mail. Then, a sub-sample of those who don’t respond are called, and some of those who don’t respond by phone are visited in-person by survey-takers. While the last steps are more costly, since they are only used for a subset of the survey participants, the multi-modal approach increases the response rate while keeping costs down. The response rate for this survey has been consistently near 98 percent of all household units.

This approach can be further refined: one possibility for the decennial Census, for example, is a promotional campaign asking people to pre-register electronically so they can receive the questionnaire by email.

Other Improvements in Survey Design Can Improve Response Rates

• Response rates improve significantly when the survey targets are contacted before the survey, with a letter or email that explains why the survey is relevant to the person being asked to take the survey.
• Multiple calls or other approaches to the person also increase response rates.
• Reducing the burden of the survey, for example by limiting the number of questions, filling in information in advance where it can be obtained from administrative sources, using a “matrix” approach so that not every person surveyed has to answer every question and using proxy reports by others who are knowledgeable when the person doesn’t respond, can all help improve the response rate and reliability of survey data.
• Financial incentives can improve response rates.

Surveys Can be Adjusted for Non-Response

Survey reliability can be improved by weighting the answers of the responses from subgroups to ensure that the sample reflects as much as possible the population being surveyed. Similar statistical measures can be used to adjust where people choose not to respond to individual questions.

Data Can be Obtained by Linking to Administrative Records

Federal administrative agencies such as the Internal Revenue Service, the Social Security Administration, the Center for Disease Control (which houses vital statistics) and the Department of Housing and Urban Development, as well as state and local agencies such as school boards and Departments of Public Health all maintain important data. If cross-linked appropriately, with great care to protect individual privacy, administrative data can provide important information that bolsters the accuracy and reliability of surveys. However, it does have limitations; for example, health records will not capture those uninsured people who do not seek care and education records won’t include the homeschooled child.

Digital Data

Digital data refers to information that is created electronically by government, business and individuals. These data might be gathered when someone makes a phone call, purchases an item, conducts a web search, or drives on an HOV lane, to give just a few examples. Commercial firms collect digital data on everything from phone records to purchasing patterns to travel records. The government also collects digital data, especially for its security and anti-terrorism programs. Some of these data, again with proper privacy protections, could be used to provide the kind of information that business, in particular, relies on.