Question

If we simply read a report of a poll, how can we know if the poll is reliable? What criteria do we use to analyze them

Answer 1

In a period of rapidly advancing technology and falling costs for computers, long-distance telephone service, and statistical software, it is easier than ever for start-up companies to get into the polling business. Because most polling now takes place on the telephone, it is cheap and easy for someone who wants to get into the polling business to buy a sample, write a short questionnaire for a Computer Assisted Telephone Interviewing (CATI) application, buy interviewing services from a field house, and receive a report based on the marginals for each question and a limited set of cross-tabulations.

Many polling organizations embrace new technology as a way to cut costs and speed data collection. Some new technologies also make it possible to collect more types of data. Web-based surveys, for example, can employ visual or audio stimuli that are not possible with other questionnaire designs, making them an excellent way to evaluate political commercials, especially when applied in a full experimental design. Many organizations have also turned to Web-based surveys to reduce the turnaround time between the design of a questionnaire and the start of data analysis and production of the first report of results.

Applied inappropriately, however, this technology offers several potential pitfalls for data quality. First and foremost are sampling issues related to respondent selection. Pollsters obtain respondents in three ways. They take "volunteers" who self-select themselves to answer generally available questionnaires on a Web site. They recruit volunteers, sometimes for a single survey and sometimes for a panel from which subsequent samples will be drawn. And they use a probability sample to select respondents on the telephone and supply Internet access to those who need it.

Because the availability of Web connections is not uniformly or randomly distributed in society, the existence of a “digital divide” can introduce one source of bias in volunteer samples. This technique, for example, tends to produce samples that are more Republican and with more conservative leanings, as we have seen in such varied circumstances as post-debate polls in 2000 and more general public policy assessments since. The resulting bias tends to favor the current Bush administration and could work against a Democratic administration. Other possible problems include fatigue from the requirement to respond to periodic and frequent surveys to maintain panel status—a requirement that could even lead, in some circumstances, to “professional” respondents. More research needs to be done on these issues, but at a minimum a poll consumer ought to know about respondent recruitment and selection for Web-based surveys.

Pollsters must also contend with the rise of cell phones. Despite the increasing penetration of these devices in the United States (approaching 75 percent), fewer than 5 percent of Americans rely solely on a cell phone. But that share is growing—and presenting pollsters with a new set of problems. First, cell phone exchanges have no general directory, and they are excluded from samples that most public polling firms can buy. Second, people who rely on cell phones are more mobile than the rest of the population, and many use phones provided by their business. If, as is likely, the geographical correspondence between the phones’ assigned area codes and their owners’ place of residence is poor, it may or may not be an issue for firms conducting surveys with national samples, but it could be for those conducting state or local surveys and effectively dialing out of their target area.

One further problem linked to new technology is telephone caller ID. This screening device, which alerts households to who is calling, makes it possible to avoid calls from “out of their area” or from unfamiliar numbers. In response to citizens’ pleas for protection from telemarketers, the federal government is moving to develop a “do not call” list. Pollsters need not honor such list membership now, but future abuses by pollsters or telemarketers could change that. This technology too is exerting downward pressure on response rates.

*The purpose of a poll is to identify how a population feels about an issue or candidate. Many polling companies and news outlets use statisticians and social scientists to design accurate and scientific polls and to reduce errors. A scientific poll will try to create a representative and random sample to ensure the responses are similar to what the actual population of an area believes. Scientific polls also have lower margins of error, which means they better predict what the overall public or population thinks. Most polls are administered through phones, online, or via social media. Even in scientific polls, issues like timing, social pressure, lack of knowledge, and human nature can create results that do not match true public opinion. Polls can also be used as campaign devices to try to change a voter’s mind on an issue or candidate.