Did Giuliani Win the Debate?

05/04/2007 01:58 pm ET | Updated May 25, 2011
  • Mark Blumenthal Mark Blumenthal is the Head of Election Polling at SurveyMonkey.

The only instant reaction survey done immediately after last
night's MSNBC/Politco Republican debate came from automated polltaker SurveyUSA.
Their release tells us, without equivocation, that Rudy Giuliani "convincingly
won" the debate, because when asked, 30% of respondents picked him as the
winner compared to 17% for McCain and single digits for other candidates. We linked
to it, as did many
other prominent
Mickey Kaus noticed, and asked a
good question:

Most commentators I've seen or heard thought Rudy Giuliani did badly in Thursday
night's debate (except my mother, who said he "makes a very good
impression.") I didn't think Giuliani did well either. So how did he
decisively win that Survey USA poll cited on The Corner? Part of the
answer is that Survey USA polled only California debate watchers.
A mere 45% of whom were Republican. 53% of whom were pro-choice.
That's not the national Republican primary electorate.

No, it's not, but an even bigger issue is the fundamental
weakness of "instant reaction" polls of this sort in judging winners and

A more general problem with using a survey to declare a
debate "winner" is agreeing on the criteria. What does it mean to "win?" Is it
enough to make a "good impression," or is the winner the one who picks up the most
new supporters? Is it about the impression made on those watching the entire
debate or on the much larger numbers who catch snippets in the news coverage
that follows? Is any coverage good coverage (think Gravel)? Do we only care
about Republican primary voters and caucus goers, or is it still important to
"win" a larger constituency. I could go on, but the various commentators seem
to apply different criteria.

What SurveyUSA did was straightforward enough. They called a
random sample of households in California
last night, and asked those who answered the phone whether they watched the
debate and, if so (13% said yes), asked who "won the debate?" It is hard to argue
with their presentation of the results. Giuliani was certainly the runaway
choice as "winner" among those who told SurveyUSA they watched it.

The problem with that approach, however, is that debates usually
serve to reinforce existing impressions. If you tuned in to last night's debate
feeling like a supporter of one of the candidates, the odds are pretty good
that if pressed, you would say that your candidate won. If most of the California voters that
watched the debate were already
Giuliani supporters, then most will likely tell us that their candidate won.

To test which candidate changed the most minds, we really
need to do what pollsters call a "panel-back" survey. They start with a very
large random sample of all adults or voters (perhaps as many as 2000) and call
several days before the debate to ask about vote preference and views of the
candidates. Immediately after the debate, they call back and ask the same
respondents (or as many as they can reach), who "won?" More important, they also
repeat the same vote and favorable ratings questions asked during the first
interview. Thus, with all the data in hand, the pollster can see which
candidate (if any) was judged a "winner" most often by those who were initially
undecided or supporting other candidates. They can easily see which candidates actually
improved their standing as a result of the debate.

Both the Gallup Organization and CBS News take exactly that
approach to polling around debates and presidential addresses, something I
wrote about many
during the 2004 campaign (see especially this

Unfortunately, the SurveyUSA poll provides only a quick
snapshot of the impressions among those who happened to be home in California last night,
but we know nothing about how respondents felt about the candidates before the debate. However, we can at
least compare the "who won" response among likely Republican primary voters to
those interviewed by SurveyUSA and other pollsters in recent weeks, as in the
following table:


First, notice that the percentage who judged Giuliani the
winner is slightly smaller among likely Republican primary voters (30%, as
above) than among all debate viewers (33%, omitted from table). But notice that
the percentage judging Giuliani the winner is smaller than Giuliani's support
in other recent polls by SurveyUSA
(43%), the Mellman
(36%), the Field Poll
(36%) and PPIC
(33%). Probably more important, 40% told SurveyUSA they thought Romney or one
of the candidates other than Giuliani and
"won" the debate last night. That is much higher than the lower tier
candidates received on the other recent surveys.

The comparison is far from conclusive. The sample sizes are
all small, and we have no way of knowing whether the sample of self-described
debate watchers was skewed to supporters of particular candidates. However, the
data we have suggests that if anything, the real winners were probably the
lower tier candidates who got a boost in exposure and recognition.

Also, put me down as skeptical that 13% of California's adult population watched the
debate last night. After last weeks' Democratic debate, Nielsen reported that 2.26
Americans watched the debate. If we assume those were mostly adults,
it amounts to roughly one percent of the roughly 219
adults in television households nationally. Perhaps this week's audience
was bigger in California,
but I doubt it was thirteen times
bigger. The 13% number is more likely is a combination of three factors: Debate
viewers were disproportionate among those who were (a) at home last night and
(b) willing to complete the interview. Also, consider the potential for (c)
measurement error -- some probably said they watched when then did not.

Typos corrected.

Update: Jay Leve,
the founder and CEO of SurveyUSA, responds in the comments section.

SurveyUSA thanks Mark Blumenthal
for his observations. SurveyUSA will use the "panel back" approach
before and after the next nationally televised debate. Those who want to
suggest other ways that SurveyUSA could improve speech-reaction and
debate-reaction polls are invited to do so.

Leve is obviously reading, so if you have a suggestion,
please leave a comment. And please bear with us if our software tells you your
comment is "waiting for approval." We have not not changed our comment policy, but have been working on some site upgrades that are behaving strangely. We will try to approve any such comments promptly.

Update II (5/6): I
had not seen it until just now, but TNR's
John Judis used the SurveyUSA data noted the same way I did on Friday and
reached a similar conclusion:

If one assumes that "who
won" tabulations are going to roughly resemble voters preferences, any
sharp divergence becomes significant. By that count, Romney was the clear
winner last night.