It was probably Murphy's Law. Within hours of my posting a
review of the sorry state of disclosure of early primary poll methodology, ABC
News and The Washington Post released
a new survey of likely caucus goers in Iowa
that disclosed the two critical pieces of information I had searched for
elsewhere. The two ABC News releases posted on the web (on Democratic
caucus results) disclosed both the sample frame and the share of the voting age
population represented by each survey. ABC News polling director Gary Langer also
devoted his online
column last Friday to a defense of his use of the random digit dial (RDD)
methodology to sample the Iowa
Let's take a closer look.
Langer concluded his column with a note on "likely voter
screening," a subject I have been posting on lately.
Some polls of likely caucus-goers, or likely voters
elsewhere, may include lots of people who aren't really likely to vote at all.
Drilling down, again, is more difficult and more expensive. But if you're
claiming to home in on likely voters, you want to do it seriously. Anyone
producing a poll of "likely voters" should be prepared to answer this
question: What share of the voting-age population do they represent?
The good news is that Langer and ABC News also provided an
answer. For the Democratic
This survey was conducted by telephone calls to a random
sample of Iowa
homes with landline phone service. Adults identified as likely Democratic
caucus goers accounted for 12 percent of respondents; with an adult population
of 2.2 million in Iowa,
that projects to caucus turnout of 260,000.
In 2004, by comparison, just over 122,000 Democrats (5.5% of
the voting age population) turned out for the caucuses.
And for the Republicans:
Adults identified as likely Republican caucus-goers accounted
for seven percent of respondents; with an adult population of 2.2 million in Iowa, that projects to
caucus turnout of 150,000. That's within sight of the highest previous turnout
for a Republican caucus, 109,000 in 1988.
The estimated turnout for the 2000 Republican caucuses was
lower (approximately 86,000), partly because John McCain focused his campaign
on the New Hampshire
primary. Thus, Republican turnout amounted to 4% to 5% of the voting age
population in the last two contested Iowa
So first, let's give credit where it is due. Of the thirteen organizations
that have released surveys in Iowa
so far this year, only ABC News has published full information about how tightly
they screened likely caucus voters.
Having said that, two questions remain: First, is the screen
used by the ABC/Washington Post poll
screen tight enough? After all, their
screen of Democrats projects to "likely voter" population of 260,000, a number more
than double both the 2004 turnout (122,000) and the all-time record for
Democrats set in 1988 (125,000). The ABC release seems to anticipate that
question with the following passage:
A more restrictive likely voter
definition, winnowing down to half that turnout, or about what it was in 2004,
does not make a statistically significant difference in the estimate --
Edwards, 28 percent; Obama, 27 percent; and Clinton, 23 percent, all within
sampling tolerances given the relatively small sample size. The more inclusive
definition was used for more reliable subgroup analysis.
The full sample had Obama at 27% and Edwards and Clinton at
26% each. While the release does not specify the "more restrictive" definition
they used, The Washington Post's version
of the results indicates that exactly half (50%) of the likely Democratic
caucus goers indicated that they are "absolutely certain" they will attend.
release makes essentially the same assertion: "A more restrictive likely
voter definition, winnowing down to lower turnout, makes no substantive
difference in the results."
So ABC's answer is: We could have used a tighter screen but
it would have made no significant difference in the results.
Their decision is reasonable considering that the Des Moines Register poll used
essentially the same degree of screening for their first
poll of Democrats in 2006, using a list based methodology that nailed the final result in 2004. Also keep in mind that no screen based on
self-reports of past behavior or future intent can identify the ultimate
electorate with anything close to 100% accuracy. Pollsters know that some
respondents will falsely
report having voted in the past, and that respondents often provide wildly
optimistic reports about their future vote intent that typically bare little resemblance to what they actually do on Election Day. And while we know what
turnout has been in the past, we can only guess as to the Iowa Caucus turnout
this coming January (or, perhaps even December). The ideal methodology defines turnout a bit more broadly than expected...[Oops, forgot to finish that sentence: An ideal method defines turnout a bit too broadly but also looks at narrower narrower turnout groups within the sample as this survey did].
The second and more complex question involves the ABC/Washington Post to use a random digit
dial (RDD) sample frame rather than a sample drawn from a list of registered
Langer makes the classic case for RDD, by pointing out the
potential flaws in samples drawn from the list of registered voters provided by
secretary of state. Roughly 15% of the voters on the Secretary of State's list lack
a telephone number and about as many will turn out to be non-working or
business numbers (according to data he cites from a Pew Research Center Iowa
poll conducted in 2003). Include the traditionally small number of Iowans that
may still register to vote (or participate after having been inactive for many
years), and we have, he writes, "a lot of noncoverage - certainly enough,
potentially, to affect estimates." Langer acknowledges that RDD samples now
face their own non-coverage problem due to the growth of cell phone only households
(12-15% now lack landline phone service), but concludes that RDD "produces far
less noncoverage than in list-based sampling."
True enough. But Langer leaves out some pertinent
information. First, campaign pollsters that make use of registered voter lists
typically use a vendor that attempts to match the names and the addresses on
the list to telephone listings. Two vendors I spoke with today tell me that they
are able to use such a process to increase the "match rate" to over 90%, a
level that makes Iowa's
lists among the best in the nation for polling.
Second - and this is a more complicated issue that really demands
another post - the potential value of sampling from a registered voter list is
not the ability to call only registered voters with the confidence that "people
are reporting their registration accurately." It also allows pollsters to use
the rich past vote history data available on the list for individual voters to
inform their decisions about which voters to sample and interview. Pollsters can
also make use of data providing the precise geographic location, party
registration, gender and age of each sampled voter provided on the list to
correct for non-response bias.
Finally, the campaign pollsters on the Democratic side that shell
to $100,000" to the Iowa Democratic Party for access to the list do not
conduct polls that "entirely exclude" first time caucus goers (as Langer
suggests). The Iowa party appends past caucus vote history to the full list of
registered voters, and pollsters can use the additional data to greatly inform
their sample selection methodology (Democrat Mark Mellman gives a hint of how
this works here;
Mellman's complete procedure probably resembles the methodology proposed by
Yale political scientists Donald Green and Alan Gerber here
Ultimately, the decision about what sample frame to use
involves a trade-off between the potential for greater coverage error (when
using a list) and greater measurement error in identifying true likely voters
(when using RDD). The decision between the two is ultimately a judgment call for the pollster. Those of us who have grown
comfortable with list samples believe that the increased accuracy in sampling true
likely voters offsets the risk of missing those without accurate phone numbers
on the lists. But the choice is not obvious. The fact that ABC and the Post
have gone in a different direction -- and have disclosed the pertinent details --
will ultimately enrich our understanding of both the poll methodology and the Iowa campaign.