The following piece is part of an ongoing series of OffTheBus reports by citizen policy experts critiquing different aspects of Campaign 08. The author is president of the American Association for Public Opinion Research (AAAPOR).
The buzz over the recent Zogby online poll that finds Senator Hillary Clinton losing to all four top Republicans in head-to-head trial heats reinforces the public's right to know and understand and the media's obligation to report the nuances of survey and polling methodology.
But what exactly should we be asking when we look at these poll results? Both the American Association for Public Opinion Research (AAPOR) and the National Council on Public Polls (NCPP) offer a list of questions for journalists to ask when reporting on polls. Consumers of election polls should be asking the same questions. One question is particularly relevant to the current debate: What population is (who are the people) represented by the poll?
In the case of an online panel survey (like Zogby's) only those individuals who have previously joined the panel have an opportunity to be surveyed. As noted in their methodology, "Zogby has assembled a database of individuals who have registered to take part in online polls through solicitations on the company's Web site, as well as other Web sites that span the political spectrum." From this pre-registered group, a random sample is selected. So what's the population that is actually represented by the poll that was recently reported by Reuters and others? The answer: those individuals who have identified themselves previously and registered to be part of the panel.
We know from the past (the 1936 Literary Digest presidential poll being the most famous) that even samples that are selected at random can be hopelessly flawed if respondents are selected from a source that does not represent the population of interest. (See here on random sampling and here for more information about sampling and why it works).
Telephone surveys are not immune to this problem (albeit for a different reason). Most telephone surveys only include land-based telephone lines, although a growing body of research has been investigating the impact of including or excluding cell phone numbers as part of the sample. Here too we can ask the question "what population is represented?" If only land-based telephone lines are included in a telephone poll, then the answer to the question is those individuals who live in residences that include a land-based telephone line. As the cell-phone-only population grows this becomes an increasingly important question to ask.
There is no one right way or wrong way to conduct pre-election polls. What all of us need to be mindful of during the primary season is that polls vary in quality--and without the disclosure of detailed methodology about the survey neither journalists nor consumers have the ability to evaluate them. Zogby in their press release does indicate that the poll of interest was an online poll--what was lost in the hype was the disclosure of that information as part of the reporting of the poll. AAPOR advocates disclosure of methodological details for all surveys and polls that are part of the public domain, and the association strongly supports the Disclosure Project of Pollster.com, an independent source that has explicitly taken on the task of soliciting this information from all pollsters and making them easily available. The Internet has rendered moot the excuse that methodological details cannot be provided due to limited space and airtime.
Disclosure Note: Mark Blumenthal, aka Pollster.com is a member of the AAPOR Executive Council
The media is doing it's masters bidding!
Ron Paul has integrity and his concern about restoring the Constitution is right on.
If only he were truly for Freedom and wasn't a forced birther .
Also the Free market does not exist . Do you really think that the Big Money , fossil fuel interests will allow alternative energy unless they can control it? They have already stopped the importation of the Smart car ( they now support it because they control it) and GMs famous electric car saga is another example.
Big Money hold all the cards in influencing government and use regulation to their advantage.
Stopping the big brother, Soviet Style of government that has grown into a monster under Bush is the #1 problem to deal with.
Without Freedom America is nothing...bush and the neoCONs , republicans and many Dems have helped make the USA a Soviet Style of government.
I'm sure that by now, everyone here is familiar with George Orwell's prophetic book - "1984". He did write other books, and in fact I had to do a book report on one of them in high school. The book is called "Animal Farm", and it can be read online at Google Books. I didn't really understand it at the time, but it is an equally prophetic book, and is well worth re-reading.
In the book, Orwell introduces one of his other famous words: "groupthink". Despite Orwell's warnings about the dangers of it, in this country, we have embraced it; although we give it a more marketable and PC name: "conventional wisdom".
Through their constant hyping of the polling data, they are able to maintain the "conventional wisdom" that HRC will win the democratic nomination, for example. Of course, when they started pushing these polls over a year ago, most people didn't even know who else was running besides Hillary, so of course these early polls favored her. In this way, they are able to take a result that is favorable to them, and through repetition and message-discipline, they are able to turn it into a self-fulfilling prophecy.
The questions asked on polls should be suspect too. Consider Hilary Clinton's recent statement that she will dedicate $50 billion to aids research if elected. Can that even be considered as there is no way she
will get the support to spend that kind of money from congress. It seems we should be looking deep into the candidates back grounds and making voting decisions based on what they've achieved rather than what they would like to achieve or what the people who have time to respond to all the questions beleave they mean when the campaign spin goes to press.
I think random nationwide phone polls are the most reliable only inso much that it is random and not a reliable number that read of subscribe to media or polls like zogby.
PollingReport is decent to scan only because it lists numerous polls , right and left based , and from there you can get a clue , but not much else.
Remember however the majority of people don't participate in polls at all.
I also am gravely concerned about the media power brokers who have made their choices about WHO are the "top tier" candidates of both parties, and upon what basis this has been done.
As a Republican who happens to be Caucasian, I can certainly tell you that Obama makes a lot of sense to me, and I could easily vote for him because I am so disgusted with the top tier Republican candidates; they only promise more of the same, and to my mind, it's not good enough. My only concern is that the rumor that he's a protege of Sen. Lieberman might be true. Also, as I look at Ron Paul I see "images" of Abraham Lincoln, and I "wonder" why the Media Moguls are down on him.
No one I know will answer a person calling them for polls for many reasons; it is invasive, it takes too much time, and it seems to result in getting more phone calls.
I wonder if there is a difference between the people who will answer polls and those who say "no thanks". I'm curious what the rate is of people who refuse to answer questions.
I also wonder if that rate has gone up since the do not call list started.
I do know this, however:
http://www.infiniteronpaul.com/meetupmaps/
thanks for the piece, nancy!