Nearly two weeks ago, just before we kicked off our Disclosure
Project, InsiderAdvantage pollster Matt Towery used a syndicated column headlined
a Quinnipiac?" to attack the Florida
polls conducted by the Quinnipiac University Polling Institute. Towery not only
highlighted how his polls differed from a recent Quinnipiac survey but also commissioned
Mason-Dixon Polling and Research to conduct a parallel poll to prove his point.
Towery's unusually jocular broadside amounted to "one pollster whack[ing] the
other upside the head," as Politico's Jonathan Martin put
"At the very least,"
Quinnipiac numbers should stop
being taken at face value as the paragon of accuracy in Florida. Somewhere in their methodology they
continue to misread the state they claim to know so intimately.
When I looked at
the four polls of Florida Republicans conducted recently by InsiderAdvanage,
Quinnipiac and Mason-Dixon, the differences between them seem explained mostly
by the inclusion of Newt Gingrich as a candidate in the Quinnipiac poll and the
fact that Fred Thompson's announcement occurred during the fielding of the
Quinnipiac poll but before the others. Still, Towery's suggestion that the
Quinnipiac differences might be found "somewhere in their methodology" led me
to ask the same kinds of methodological questions as we have been asking as
part of our Disclosure Project. Their responses follow, and the difference in
sampling methodology adds another possible explanation. Quinnipiac's sample of
"registered Republicans" samples a population roughly four times the size of
the "likely voters" surveyed by InsiderAdvantage and Mason-Dixon.
Interview Dates -
One question I asked only of Quinnipiac was to provide the number of interviews
conducted before and after Fred Thompson's announcement of candidacy. Doug
Schwartz at Quinnipiac reports that 199 (or 45%) of their 438 interviews were
conducted on or before the evening of September 5. Thompson declared his
intentions later that night and received a burst of positive coverage in the
week that followed. While the methodologies of these surveys differ, it is
worth remembering that the other polls by InsiderAdvantage and Mason-Dixon were
fielded in their entirety after September 5.
Sample Frame -
Although the term is a bit wonky, one of the most important ways these polls
differ is in what pollsters call the "sampling frame." Put more simply, the
issue is the source for the random sample of voters called by each pollster.
Quinnipiac uses a random digit dialing (RDD) methodology
that contacts a random sample of all the working landline telephone numbers in Florida and then uses
screen questions to select a random sample of registered Republicans. In this
case, both InsiderAdvantage and Mason-Dixon selects voters at random from the
list of registered Republicans provided by the Secretary of State, using both
actual vote history and screen questions to identify and interview "likely"
Republican primary voters.
How Did They Select Republican
Registered or Likely Voters? - The pollsters at Quinnipiac provided a
complete and relatively straightforward answer. They asked two questions about
vote registration and party affiliation:
Some people are registered to vote and others
are not. Are you registered to vote in the election district where you now
live, or aren't you?
[IF REGISTERED] Are you registered
as a Republican, Democrat, some other party or are you not affiliated with any
Both Towery and Brad Coker at Mason-Dixon were both
initially reluctant to describe the specifics of their likely voter selection
procedures, citing the need to protect "proprietary" methods. After a bit of email
back-and-forth, however, both were willing to describe their methods in general
terms. Let's start with Coker, answering on behalf of Mason-Dixon:
Our sample design and screening method takes
into account voter registration, party registration, past primary voting
history and likeliness to vote in the primary. Other factors that were taken
into account were the age, county, gender and race of the voting population
based on previous Republican primary elections.
What that means -- as I read it -- is that Mason-Dixon uses
information on past vote history on the voter list to draw a random sample of a
subset of Republicans that they consider most likely to vote. They ask those
sampled individuals questions on "likeliness to vote in the primary" and screen
out unlikely voters. Finally, they weight the demographics of the final sample
based on the demographics of voters in previous Republican primaries.
Towery reports using a similar procedure at InsiderAdvantage:
"We do poll off of [a list of] registered voters, but we do then cull that
number down based on a voting history that gives us a more likely voter
sample." They then ask a screen question to identify likely primary voters:
"are you likely to vote in the_____presidential primary to be held ____."
What Percentage of
the Voting Age Population Did Each Poll Represent? - The calculation for
the Quinnipiac poll is relatively straightforward: They report starting with a
random sample of 1,325 adults and using the questions above to identify and
interview 438 registered Republicans. So the Quinnipiac Republican sample
amounted to 33% of Florida adults (438 divided by 1,325).
Again, Coker and Towery were initially reluctant about
sharing specific numbers, but ultimately provided the information necessary to
answer my question. Let's start again with Coker and Mason-Dixon:
The population we were trying to capture was
the roughly 1 million Republican voters who will be most likely to vote in
January. In a universe of approximately 3.8 million registered Republicans, we
targeted a population of about 1.2 million Republican voters and had an
incidence of 83%.
interviewed a sample designed to represent approximately 1 million voters (1.2
million * .83) out of 14.2 million Florida
adults, or 7.0% of Florida adults.
Next, Towery and InsiderAdvantage:
The [target] universe based on our sample
system was around 1.6 million. This reflects the slightly higher than normal
turnout you see in a Presidential primary. Incidence rate, based on data I just
received was around 75%. Based on your description this would mean a final
"universe" of around 1.2 million voters (all registered) which I believe
reflects the likely turnout for a GOP Pres. primary turnout.
Insider-Advantage interviewed as ample designed to reflect approximately 1.2
million adults, or 8.5% of Florida adults.
As should be
obvious, Mason-Dixon and InsiderAdvantage sampled significantly narrower
populations of Republican voters than Quinnipiac. Via email Brad Coker argues that a narrower "screen" is more appropriate to a
pre-election poll aimed at projecting the preferences of likely voters: "I
would question the validity of a poll of ‘registered Republican voters' simply
on the grounds that 75% of those sampled probably won't be voting in January."
According to the Florida
Secretary of State, the vote for Republican primary candidates totaled
roughly 1 million in 2006 (Governor), 1.2 million in 2004 (Senate) and 700,000
in the 2000 presidential primary.
In response to my
initial questions, Quinnipiac's Doug Schwartz sent this statement:
The methodology used by the Quinnipiac poll is similar to
that of all the other major polling operations in the country. It has correctly
predicted the outcome of every major race it has polled on in Florida during the past three years. For
details on the methodology used, contact Doug Schwartz or visit http://www.quinnipiac.edu/x271.xml
That is true, but we should note that the final pre-election survey conducted by Quinnipiac in 2006 reported the
results among "likely voters" rather than all registrants. So their
primary voter "methodology" may shift as we get closer to
Election Day. Pollsters continue to debate the merits of various likely voter
models months prior to the election, something I covered in great
detail in 2004 in the context of general elections. Putting that debate aside,
however, the point is that the universes sampled in this instance are very different.
What Are the Demographics? - I asked
each pollster to provide the results to demographic questions asked of their
Republican samples. Both Quinnipiac and Mason-Dixon were quick to respond. The
table below shows that the Quinnipiac sample is a bit younger. This is not
surprising given that voters are typically older than non-voters.
and Mason-Dixon also included the regional composition of their samples. While
their regions were not identical, their definition of South
Florida came close. Mason-Dixon had fewer Republicans in their
Southeast Florida region (18% in Palm Beach, Broward, Dade and Monroe Counties)
than Quinnipiac (23%; although the Quinnipac South Florida region also includes
Hendry County, which accounts for just 0.1% of registered Republicans
difference is important because, according to the Mason-Dixon cross-tabulations
that Coker also provided, Rudy Giuliani ran far ahead of Fred Thompson (33% to
11%; n=70) in Southeast Florida, but trailed Thompson narrowly elsewhere (22%
to 26%; n=330). So the fact that Quinnipiac had a greater percentage of respondents
in South Florida provides yet another explanation
for Giuliani doing better statewide in their poll.
provided this information:
Since we have the Florida voter file, we know the precise
demographic profile of those who have voted in previous elections (at least in
terms of county, age, gender and stated race/ethnicity). Our sample matches it
within 1-2% of the actual figures from the average of 2004 & 2006 GOP
primary turn-outs. Deaths and out-migration could easily account for any
Towery, on the
other hand, was more reticent:
We don't give out our weighting percentages or
our demographic regional breakdowns because those are proprietary and if we did
so, it would be like Coke giving away the secret formula, well not that big,
but important to us!
Which brings us
back to the whole point of our Disclosure Project. We should congratulate all
three pollsters for providing the "incidence" data necessary to help us answer,
in essence, not just "What is a Quinnipiac?" (to borrow Towery's headline) but
also, "what is a Mason-Dixon?" and "what is an InsiderAdvantage? As a result of
their disclosure, we can see how different the "target populations" were and
take those differences into account in assessing the results.
It took some
coaxing, to be sure. Coker has previously
refused similar requests on the grounds of protecting proprietary interests.
Given his extensive experience in Florida (he
tells me he has conducted more than 200 statewide polls in Florida since 1984), Coker was
understandably reluctant about responding in this instance. So his cooperation
here is noteworthy. Hopefully, other pollsters follow his lead because the
general descriptions and incidence calculations provided above could be easily
replicated by every pollster and released online for every poll. Similarly, a
demographic composition table, like the one above, would be an easy addition to
the online documentation virtually every pollster and news organization makes
available for every poll.
On the other hand,
Towery's "secret formula" dodge has a fundamental flaw. Coke need not give away
its "secret formula" when it prints on every can, as required by law, a list of ingredients, the number of calories and the grams of carbohydrate and other
nutrients contained in each serving. As should be obvious, the our
Constitution's First Amendment precludes the sort of mandatory labeling for pollsters that the
FDA requires for food. However, pollsters like Towery ought to start thinking
about how to better label their own products in terms of their sample
composition, lest some snarky blogger ask, "What's an InsiderAdvantage?"
How will Donald Trump’s first 100 days impact YOU? Subscribe, choose the community that you most identify with or want to learn more about and we’ll send you the news that matters most once a week throughout Trump’s first 100 days in office. Learn more