THE BLOG

Kentucky Governor: Undecided or Still Trying to Decide?

07/19/2007 06:56 pm ET | Updated May 25, 2011

The latest automated SurveyUSA poll in the Kentucky
Governor's race provides us with one of those classic conflicting poll stories
that we just love here at Pollster.com, because it illustrates how small differences
in methodology can have a profound effects on the results. In this case, SurveyUSA
shows Democrat Steve Beshear leading incumbent Republican Ernie Fletcher by a
23 point margin (59% to 36%) with only 5% undecided. Meanwhile, an InsiderAdvantage
poll conducted a week earlier shows Beshear leading by just three points (41%
to 38%) with a much larger number (21%) in the undecided category

What explains the difference? Continue after the jump for
more explanation, but my best guess is that the solution can be found in this
conundrum: On a poll, "undecided" means something different than "still trying
to decide."

The results shown below add a little more context. Back in
May, SurveyUSA showed Beshear leading by an even wider (62% to 34%) margin at
about the same time that another automated survey by Rasmussen Reports showed
Beshear 16 points ahead (51% to 35%).

07-19%20Ky%20Gov%20surveys.png

One obvious difference is that SurveyUSA and Rasmussen use
an automated, recorded voice that asks respondents to answer by pressing
numbers on their touch tone phones. InsiderAdvantage uses live interviewers.

As part of a presentation I did with Charles Franklin at the
AAPOR conference in May, I took the statewide surveys we gathered during 2006
and calculated the average undecided/other response for automated and
conventional (interviewer) phone polls during each month (technically, I
averaged results within states, and then averaged values for all states,
looking at what was left after summing results for each reported candidate). The
bottom line is that the undecided percentage was typically 3 to 5 percentage
points lower on automated polls than those using live interviewers.

07-19%20undecided%20ivr%20phone_small.png

The key question about this pattern, as suggested by the chart's
subtitle, is whether it results from IVR sampling different kinds of people or from respondents giving different answers on an automated survey.

It is certainly possible, of course, that kinds of people
willing to participate in an automated sample might be more opinionated than
those who needed extra prodding from a live human being. Critics of the
automated polls frequently make this argument, but this is a tough claim to
evaluate given the data we have available.

It is also possible that other sampling differences - such
as the sample "frame" and the kinds of voters that qualified as "likely" -
helped produce the different result in this example from Kentucky. SurveyUSA reports, for example,
that they started with a random digit dial (RDD) sample of 1,000 adults and
screened down to the 560 they deemed most likely to vote.

InsiderAdvantage
reports that they interviewed "693 registered voters." Theoretically, those who
say they are likely to vote should be a bit more opinionated than those who are
just registered, a factor that may contribute to the difference in the
undecided percentage. InsiderAdvantage provides no further information about
their sample, although they did tell me
in October 2004 that all of their surveys that year sampled from registered
voter lists rather than RDD.

In this case, however, my guess is that the most likely culprit
is not the sample but rather the way the two polls measured vote preference,
and more specifically, how hard they pushed voters to express a preference.

Automated pollsters argue that they get a lower undecided partly
because the Interactive Voice Response (IVR) methodology better simulates the
secret ballot. Respondents are sometimes reluctant to reveal their preference
to a stranger on the telephone, hesitating to voice an opinion might introduce
some "social discomfort" into the conversation (for either the respondent or
the interviewer). Conventional pollsters have long appreciated this challenge,
which is why very few provide "undecided" as an explicit answer category. Interviewers
are usually trained to record "undecided" as an answer only when respondents
cannot provide an answer.

Most political pollsters also include a follow-up question
that presses the initially reluctant respondents about which candidate they
"lean" toward supporting. SurveyUSA adapts this practice to their automated
methodology by offering "undecided" as the final answer option, but only after
a pause of several seconds that essentially pushes respondents to make a
choice.

All of this makes the following webcast
comments of InsiderAdvantage CEO Matt Towery especially telling:

When you are this far out from an
election, November is the election day, you have
to
make sure that the individuals who are taking the poll understand that
they have the option to say "undecided" and that's very important when you
survey this far out in a gubernatorial race, particularly one that's in flux.

If InsiderAdvantage explicitly prompted for undecided, it is
not at all surprising that their survey produced a much higher than average
undecided percentage than the SurveyUSA poll.

So which approach is right? Virtually all pollsters agree
that it is poor practice to suggest "undecided" as an answer a few days before an election, a practice that will likely result in a less accurate forecast. However, this issue is not as
clear four months before an election when relatively few voters have made a
final decision. Towery is right to argue in his webcast that "large numbers of
people . . . don't care about politics until the bitter end." This far out, voters may express
a preference when asked to choose, but that preference is still subject to change.

As evidence, consider the results from this week's WMUR/CNN/UNH New
Hampshire primary poll
. Roughly nine out of ten Republicans (88%) express support for one of the candidates on the trial heat question, but only 7% have "definitely decided who they
will vote for," and 71% of New Hampshire Republicans are "still trying to decide." Similarly,
91% of Democrats in New Hampshire
express a preference, but only 10% are certain and 64% are still trying to
decide.

This is the key point: Expressing a preference on a pre-election
poll months before an election is not the same as making a final decision. If
you assume that the "undecided" category captures everyone still weighing their
choices, you will badly misinterpret the results whether you think the better
"undecided" number is 5% or 21%
.

The most striking thing to me in the Kentucky results is the relative consistency
in support for Fletcher across all four polls. Fletcher's support varies within
a four point range (34% to 38%), while support for Beshear varies widely (from
41% to 62%). That pattern is not unusual in
a race involving a well known incumbent, particularly when the incumbent is embattled
by various scandals that have lowered his job performance rating to just
39%
of adults. Attitudes about the incumbent typically drive voter
preference at this stage of the race, lesser known challengers typically gain
support as the campaign progresses.

Collectively, these results suggest trouble ahead for
Governor Fletcher unless he can convince a big chunk of Kentucky voters to either reconsider his
performance as governor or rule out Steve Beshear as a credible alternative.

Finally, for what it's worth, my preference even at this
stage in a race is to push initially hesitant respondents hard to express a
preference, but then to immediately follow-up with a question (like the one
asked by WMUR/CNN/UNH) about how certain they are to actually support their
preferred candidate. Every campaign pollster I know asks a variant of the "certainty"
question -- I wish more media pollsters would do the same.

PS: One more thing. The SurveyUSA question includes the
names of the two running mates, while InsiderAdvantage appears to name just
Fletcher and Beshear. While Steve
Beshear
held statewide office in the 1980s and was the Democratic nominee
for U.S. Senate in 1996, his running mate Dan Mongiardo lost an
even closer Senate contest just three years ago. So the ticket of Beshear and
Mongiardo may sound like a more compelling alternative to uncertain voters than
Beshear alone.