Tonight,
partly in response to a critique
offered here in May, SurveyUSA fielded an automated "panel-back"
survey among 717 Americans who said they watched tonight's CNN/YouTube
debate. The idea of a panel-back survey is to interview the same sample before
and after some event, allowing for an individual level measurement of change. With
a panel back survey, a pollster can determine how many respondents shifted
their position
Here is the
key question asked of these respondents before and after the debate:
Regardless of who you may vote for,
and regardless of whether or not you would vote for a Democratic candidate for
President, which Democratic candidate would make the best President of the United States?
The result:
Americans who watched the YouTube Democratic Debate 7/23/07
on CNN went into the debate thinking Hillary Clinton would make the best
President of the 8 Democrats on stage, and came out of the debate even more
convinced, according to an exclusive SurveyUSA poll. Before the debate, 40% saw
Clinton as the
most Presidential Democrat; after the debate, when the same respondents were
re-interviewed, 43% saw her as the most Presidential Democrat
It is not
obvious from the before
and after snapshot, but it appears that most of those who responded to both
calls were consistent in their answer to this question, before and after. To
the extent that there was a shift, however, it worked slightly to the benefit
of Clinton and Biden, slightly against Barack Obama.
They also
asked if respondents "viewed each Democrat positively or negatively" before and
after the debate. The results indicate that exposure to the debate improved
perceptions of all the candidates except for Gravel. The summary reported by
SurveyUSA (with bullets added):
- Bidenwent up 38 points, from Minus 6 to Plus 32.
- Obamawent up 17 points, from Plus 24 to Plus 41.
- Clinton went up 16 points, from Plus 34 toPlus 50.
- Edwardswent up 16, from Plus 22 to Plus 38.
- Doddwent up 15, from Minus 21 to Minus 6.
- Richardson went up 14, from Minus 1 to Plus13.
- Kucinichwent up 7, from Minus 21 to Minus 14.
- Gravelwent down 3, from Minus 29 to Minus 32
The SurveyUSA
summary
does include some important caveats worth noting. The most important is that
this sample of 717 is not a random sample of all Americans (or likely voters),
or even a random sample of all Americans who happened to be at home this
evening. They started with a first round of calls to a random sampling of
households, but this final pool of 717 reflects only those who responded to the
first survey, reported that they would watch the debate and take a second call and
then actually completed a second call. The summary also notes that "panel
back is not universally sanctioned" as a tool to measure debate reaction,
pointing to the unsparing criticism by pollster David Hill. Writing for The
Hill in 2004, he described the panel-back as "worthless:"
Considerable scholarly research
demonstrates that simply being interviewed renders an otherwise normal voter
abnormal. After being polled, voters are much more likely to seek out political
information through the media, discuss politics with others and eventually to
vote. The known effects are so great that in the earliest days of polling,
voters would be screened at the outset of an interview to ascertain if they had
ever been interviewed before.
I don't share Hill's blanket aversion to panel-back surveys,
but he is right to warn that the first interview can add some artificiality to
either the nature of the sample or the way those surveyed watch and react to the
debate. For all its drawbacks, however, this approach is a great improvement to
simply calling a fresh sample of those at home and asking "who won" without any
data on which candidate they supported beforehand.
In the spirit of better understanding the data, I do have a
few questions for the analysts at SurveyUSA:
First, what percentage of the 717 debate watchers provided
the same answer to the first question before and after the debate?
Second, the summary reports:
SurveyUSA dialed at random into all 50 states. Respondents
were asked if they planned on Monday night, 7/23/07, to watch the entire
YouTube debate on CNN. Those respondents who said "yes" were then
asked if SurveyUSA could call them back, immediately after the debate
What percentage of those adults interviewed on the first
call agreed to watch the debate
and be interviewed a second time?
Finally,
did SurveyUSA have any way to confirm that they interviewed the same individuals
within each household before and after the debate?