It's probably the blogger variant of Murphy's Law, but the
most interesting topics often bubble up whenever I take time off. The last few weeks were no exception as
several new polls were released on the 2008 primary contests, particularly in Iowa and New
often happens this early in the process, some produced contradictory results,
especially in Iowa.
The most puzzling - as noted by our
friend Mickey Kaus - involves the performance of Hillary Clinton and Barack
Obama in two polls of likely Democratic caucus goers conducted in Iowa in late December by
Research2000 and the American Research Group (ARG). Both showed John Edwards with roughly the
same support (20-22%). ARG
showed Clinton leading with 31% and Obama
running distant forth (at 10%) behind outgoing Iowa governor Tom Vilsack (17%). Research 2000
ARG showed Obama and Edwards tied for first
(22%), with Clinton
running forth (10%) behind Vilsack (17%).
So...Hillary Clinton is either their clear front runner in Iowa (with 31%) or running
a distant fourth (with 10%).
Ladies and gentlemen, welcome to the pollster's nightmare: The Iowa Caucuses.
I have written before about the challenge of polling the caucuses
before and will certainly do so again, but the numbers behind the challenge
remain the same. Here is the way I put
it, when the Des Moines Register
released its first 2008 caucus poll last June:
The big challenge for polling this
contest, of course, is that turnout for the Democratic caucuses is typically a
small percentage of eligible voters. Iowa had roughly 2.2 million voting eligible adults in
2004, of whom (as of last month) approximately 1.9 million are considered "active"
registered voters by the Iowa Secretary of State. But only 124,331
participated in the 2004 Democratic Caucuses for President (according to the
subscription only Hotline).
That number amounts to roughly 6% of all registered voters, so selecting
"likely caucus goers" is no easy task.
When I first saw the conflicting results, I assumed
something obvious about the survey design or field dates might explain the
difference. For example, some pollsters
sample likely caucus-goers by calling a random digit dial (RDD) sample of all
telephone households and will then screen for likely voters. Some will sample from the lists of registered
voters (with many unlisted numbers missing) and select using a combination of
screening and various "vote history" criteria, including participation in past
caucuses. In past elections, Iowa caucus surveys
drawn exclusively from lists of past caucus-goers have differed from those
based on RDD methods.
I spoke earlier today to both Dick Bennett of ARG and Del Ali
of Research 2000, and in this case the sample procedures and field dates were
more similar than different:
- Research 2000 conducted a survey among 400 Democratic "caucus goers" December 18 through December 20. They started with a random digit dial (RDD) sample of Iowa households and screened for those who (a) say they frequently vote in statewide general election[s] and (b) report having participated in the 2004 Democratic caucuses.
- The American Research Group conducted a survey among 600 "likely Democratic caucus goers" between December 19 and December 23. They too started with a random digit dial (RDD) sample of Iowa households and screened for those who were (a) registered vote as either Democrats or with no party affiliation who also said they (b) "definitely plan to participate in the 2008 Democratic presidential caucus" (those screened out those who said they "might" participate or who "probably will not").
The field dates overlap, so timing seems unlikely to explain
much of the difference, particularly if the theory is that Obama's support has
been rising of late. Keep in mind that
the ARG poll, which finished later, showed Obama doing worse.
The biggest difference that we are aware of is that the Research2000
screens were based on self-reported past
participation, while ARG screens rely on a question of prospective intent to participate.
We can debate the relative merits
of each approach (and no doubt will in the coming months), but it seems
unlikely that this particular difference produced a 21-point shift in support
for Hillary Clinton.**
Of course, it may be that one approach was significantly
"tighter" than the other. That is, did
one capture a much narrower slice of Iowa
voters than the other? Unfortunately,
neither pollster has released data on how many otherwise qualified respondents
they screened out in order to select their final sample (as the Des Moines
Register did last June).
It is also important to focus on the questions asked. Both pollsters asked respondents to choose
from a list of eleven potential candidates, both "rotated" (or randomized) the
order of names as read by interviewers and both reported relatively few in the completely
undecided category (11% for Research 2000 and 8% for ARG). But the candidate listings were not
identical. ARG included the names of two
potential candidates -- Connecticut Senator Chris Dodd and former Alaska
Senator Mike Gravel - that Research 2000 omitted. Likewise, Research 2000 included Al Gore and
Evan Bayh, while ARG did not. Of these,
Gore had the most support (7%), Dodd had 2% and the rest just 1%. Given the numbers involved, it is hard to see
how these minor differences contributed much to the Clinton-Obama discrepancy.
All of which leaves me scratching my head, except to say
this: Whenever very small differences in
methodology make for huge differences in results, it suggests that voters are
not yet engaged in the race enough to have strong allegiances. Put another way, while each poll may have a
candidate running in front, in Iowa
at least, there is not yet a true "front runner."
**UPDATE: Ok, make that coming hours. Mickey Kaus considers the prospective / retrospective difference in the survey screens a more likely explanation than I did:
There's a big difference between 1) asking voters if they "definitely plan" to go to the caucuses, and 2) asking voters if they actually participated in the 2004 caucuses. Lots of people say they "plan" to attend. That's normal! But those who have attended are the sort of pathetically unrepresentative hard core activists ...sorry, committed citizens who make up the tiny sliver (6%) of Iowa voters who actually show up and choose the winner: ... In this case, the merely aspirational caucusgoers pick Clinton, while the hard core goes for Obama--a result consistent with the idea that Obama is capturing those who think a lot about politics, while those who don't think as much about politics haven't yet been hit by the wave.
That's a plausible theory, particularly if the retrospective caucus participation question successfully identified actual past caucus goers. Retrospective vote questions typically over-report past voting behavior, but in this case the Research 2000 question may have produced an appropriately tighter screen. Of course, without the ability to compare the relative incidence of each survey, we are just speculating.
In 1988, I worked for Paul Maslin, the pollster for Democratic Senator Paul Simon. Simon always did better on the samples we drew from lists of actual past caucus-goers, while Congressman Dick Gephardt did consistently better when when we included registered voters that had not participated in the previous caucuses in 1984. Gephardt also did consistently better on the RDD surveys in the public domain. As I recall, those differences persisted through the final round of polling, though they probably narrowed a bit toward the end. Of course, the challenge is that every election year, the caucuses attract large numbers of voters who did not participate in the prior election cycle. And true junkies will remember that Gephardt ultimately won the Caucuses, although as I recall, the actual result fell somewhere between the two methodologies.
Follow Mark Blumenthal on Twitter: www.twitter.com/MysteryPollster