Pollster reader Petey posted a critical and incisive comment to my post
last week showing that a new Harris Interactive Internet panel survey produced
ratings of Hillary Clinton that were not wildly inconsistent with other recent conventional surveys:
Am I missing something, Mark?
Just because the Harris results are somewhat
in line with random sampling poll results doesn't mean they should be treated
as a real data point.The Zogby Interactive results in the last election
likewise were somewhat in line with other results, but they were still basically
un-useful as polls.No matter how good the weighting, self-selected
polling is always going to be fundamentally flakey, no?
Petey -- assuming this is the same guy who often leaves astute
comments on Pollster and Mystery Pollster -- does not miss much. His pointed
question made me realize that a few lines of that post could have been written
better. Specifically, by writing that a side-by-side test of the Clinton item using both
the Internet panel and a traditional phone survey might "help resolve the sampling
question," I implied that such a test might establish the validity of the
Harris method. What I meant, more narrowly, as that it might resolve whether
the Harris method produced a sample more hostile to Hillary Clinton in this
instance (although since that post we now have another data point that allows a
better comparison - see below). The larger issue of the merits of Internet
panel surveys will not be resolved by any one test or blog item.
The problem with Internet panel surveys generally is their
departure from random sampling. Traditional surveys begin by drawing a random
sample from a "frame" (the pool of all wired telephone numbers, for example)
that allows all or most of the population of interest a chance of being
selected. Internet panels draw their samples from a pool of individuals who
have volunteered to participate in online surveys, usually by responding to a
banner advertisement. That fundamental difference, to answer Petey's question,
is something that should make us more skeptical of Internet panel surveys
generally.
However, it is important to remember that traditional
telephone surveys now face fundamental challenges of their own. Response rates
have fallen to below 30% on the best of public polls (and far lower on many
others in the public domain), while the rapid growth of mobile phone usage has
reduced the coverage of wired random digit dial (RDD) telephone samples below
90%. Since the true "science" of random probability sampling requires the
assumption of 100% coverage and response - a goal that no public opinion poll
comes anywhere close to reaching - we have to realize that any survey is now theoretically "flakey."
So how do we evaluate the relative "flakiness" of the competing
methods? When it comes to non-probability Internet samples, some of my
colleagues in the American Association for Public Opinion Research (AAPOR) like
to remind me that (as one put it on Friday), "we can't accept a survey as valid just because we like the
results." In other words, echoing Petey's point, if a survey lacks a sound theoretical basis, it should not be
trusted just because it produces results that are consistent with other polls.
The problem with that argument, unfortunately, is that our
continued trust in traditional surveys (despite their fundamental flaws) rests on studies that evaluate the
results. We find reassurance in the way conventional surveys continue to
predict election outcomes about as well as in past years. And the most rigorous
academic studies -- such as those I saw presented at a workshop sponsored last
Friday by the Washington DC chapter
of AAPOR -- find few examples of bias due to low response rates once they are weighted
demographically. For example, the influential study
conducted by Scott Keeter and his colleagues at the Pew Research Center that
used a side-by-side experiment to compare their standard methodology to a more rigorous
design that produced a much higher response rate. The result? "[W]ithin the
limits of experimental conditions, non-response did not introduce substantial
biases in the estimates."
In other words, despite fundamental theoretical flaws in our
traditional methods, we still like the results.
As such, those who produce surveys based on non-probability
samples are right to ask whether their methods should also be evaluated on
their "track record" (as Harris Interactive's Humphrey Taylor argued in an
article in the January 15 print edition of the Polling Report). And comparing
track records is what we aim to do here at Pollster.
The Zogby "Interactive" polls released in 2006 suffered from
more than just theoretical questions Their results were far more
variable and less
valid (when compared to election results) than other conventional polls. And
the public Internet panel surveys conducted during the 2004 U.S. presidential
campaign by all sources (Zogby, Harris and YouGov) all showed a small but a
consistent bias toward the Democrats (Professor Franklin and I been crunching
the numbers on 2004 and 2006 and will have some posts to share on this subject
very soon).
But all Internet panel surveys are not created equal, and
past performance may not be a guarantee of future results - for any survey
method. All of which brings me back to the Harris poll on Hillary Clinton. Since
my post last week, Time released its latest
survey which, by chance, included a question with a more similar wording
and structure to the Harris item than those other surveys I looked at last
week.
Time/SRBI - [Asked of 1,102
registered voters only, March 23-26] If the following candidate were to run for
president and the election was being held today, how much would you support
this candidate, definitely support, probably support, probably not support
definitely not support?Harris Interactive -
[2,223 adults via Internet panel, Marc 6-14] If Hillary Clinton was the
Democratic nominee for President, which is closest to the way you think? I
definitely would vote for her, I probably would vote for her, I probably would
not vote for her, I definitely would not vote for her, I wouldn't vote at all
In this case, the Time survey shows a bigger number (46%) in
the two support categories than Harris (37%), although two big differences
between the measurements (other than sampling) remain. The first is that the
Harris item offered respondents the explicit choice of not voting at all, while
the Time question offered only four categories of support. The second is that Time
asked the question of registered voters only, while Harris asked it of all
adults. We could get a closer comparison if we could tabulate the Harris result
among self-reported registered voters, but no such results appear in the Harris
release.
Nonetheless, we have at least a hint here that those who
volunteer to participate in Internet panels may be less supportive of Hillary
Clinton than respondents interviewed by conventional methods. And that is
something we need to keep an eye on.
[Update - Interests disclosed: The primary sponsor of Pollster.com is the research firm Polimetrix, Inc. which conducts online panel surveys].