The following piece is part of an ongoing series of OffTheBus reports by citizen policy experts critiquing different aspects of Campaign 08. The author is president of the American Association for Public Opinion Research (AAAPOR).
The buzz over the recent Zogby online poll that finds Senator Hillary Clinton losing to all four top Republicans in head-to-head trial heats reinforces the public's right to know and understand and the media's obligation to report the nuances of survey and polling methodology.
But what exactly should we be asking when we look at these poll results? Both the American Association for Public Opinion Research (AAPOR) and the National Council on Public Polls (NCPP) offer a list of questions for journalists to ask when reporting on polls. Consumers of election polls should be asking the same questions. One question is particularly relevant to the current debate: What population is (who are the people) represented by the poll?
In the case of an online panel survey (like Zogby's) only those individuals who have previously joined the panel have an opportunity to be surveyed. As noted in their methodology, "Zogby has assembled a database of individuals who have registered to take part in online polls through solicitations on the company's Web site, as well as other Web sites that span the political spectrum." From this pre-registered group, a random sample is selected. So what's the population that is actually represented by the poll that was recently reported by Reuters and others? The answer: those individuals who have identified themselves previously and registered to be part of the panel.
We know from the past (the 1936 Literary Digest presidential poll being the most famous) that even samples that are selected at random can be hopelessly flawed if respondents are selected from a source that does not represent the population of interest. (See here on random sampling and here for more information about sampling and why it works).
Telephone surveys are not immune to this problem (albeit for a different reason). Most telephone surveys only include land-based telephone lines, although a growing body of research has been investigating the impact of including or excluding cell phone numbers as part of the sample. Here too we can ask the question "what population is represented?" If only land-based telephone lines are included in a telephone poll, then the answer to the question is those individuals who live in residences that include a land-based telephone line. As the cell-phone-only population grows this becomes an increasingly important question to ask.
There is no one right way or wrong way to conduct pre-election polls. What all of us need to be mindful of during the primary season is that polls vary in quality--and without the disclosure of detailed methodology about the survey neither journalists nor consumers have the ability to evaluate them. Zogby in their press release does indicate that the poll of interest was an online poll--what was lost in the hype was the disclosure of that information as part of the reporting of the poll. AAPOR advocates disclosure of methodological details for all surveys and polls that are part of the public domain, and the association strongly supports the Disclosure Project of Pollster.com, an independent source that has explicitly taken on the task of soliciting this information from all pollsters and making them easily available. The Internet has rendered moot the excuse that methodological details cannot be provided due to limited space and airtime.
Disclosure Note: Mark Blumenthal, aka Pollster.com is a member of the AAPOR Executive Council