A suggestion from alert reader and frequent commenter
Andrew:
I write to suggest that you analyze
the huge discrepancy between the latest Rasmussen and Washington Post/ABC
polls. I'm talking about the Republican nomination. Rasmussen says Thompson is
up by 4 over RG, while WP/ABC says Rudy is up by 20 pts over FT, who isn't even
in second place here (36 RG to 14 FT). One of these pollsters is
obviously very wrong. Two polls cannot both be accurate, if their margin of
victory do not approximate each other. This is a humongous 24 point
discrepancy.
Here, with a little assist from Professor Franklin, is a
chart showing the discrepancy that Andrew noticed. The two surveys do seem to
show a consistent difference that is clearly about more than random sampling
error. The ABC News/Washington
Post survey shows Giuliani doing consistently better, and Thompson
doing consistently worse, than the automated surveys conducted by Rasmussen
Reports, although the discrepancy has been largest in terms of how the most recent ABC/Post poll compares to Rasmussen surveys conducted over the last month or so.
To try to answer Andrew's question, it makes sense to take
two issues separately. First, why are
the surveys producing different results for the Republican primary?
At the most basic level, these surveys seem to be measuring
the same thing: Where does the Republican nomination contest stand nationally? And
both surveys begin with a national sample of working telephone numbers drawn
using a random digit dial (RDD) methodology. Take a closer look, however, and
you will see some pretty significant difference in methodology:
- TheABC/Post survey uses liveinterviewers. Rasmussen uses an automated recorded voice that asksrespondents to enter their answers by pushing buttons on a touch tonekeypad. This method is known as Interactive Voice Response (IVR). Theresponse rates -- and more importantly, the kinds of people that respond --are likely different, although neither pollster has released specificresponse rates for any of the results plotted above.
- TheABC/Post survey attempts toselect a random member of each household to be interviewed by asking "tospeak to the household member age 18 or over at home who's had the lastbirthday" (more details here). Rasmussen interviews whatever adult memberof the household answers the telephone. Both organizations weight thefinal data to reflect the demographics of the population.
- RasmussenReports weights each survey byparty identification, using a rollingaverage of recent survey results as a target (although their partyweighting should have little effect on a sub-group of Republican primaryvoters). The ABC/Post surveydoes not weight national surveys at this stage in the campaign by partyID.
- [Update -- one I overlooked: The ABC/Post survey includes Newt Gingrich on their list of choices. Gingrich receives 7% on their most recent survey. If the Rasmussen survey prompts Gingrich as a choice, they do not report it.
It is also possible that Rasmussen omits other candidates as well, as tTheir report provides results for just Giuliani, Thompson, Romney and McCain. Update II -- Scott Rasmussen informs via email: "We include all announced candidates plus Fred Thompson"].
- Andperhaps most important for Andrew's question: The ABC/Post survey asks the presidential primary question of alladults that identify with or "lean" to the Republicans. The Rasmussensurvey screens to a narrower slice of the population: Those they select as"likely Republican primary voters."
Unfortunately, neither pollster tells us the percentage of
adults that answered their Republican primary question, but we can take a
reasonably educated guess: "Leaned Republicans" have been somewhere between 35%
and 42% of the adult population on surveys conducted in recent months by Gallup and the Pew
Research Center. If Rasmussen's likely voter selection model for Republican
is analogous to their model
for Democrats, their "likely Republican primary" subgroup probably
represents 20% to 25% of all adults.
Consider also that, even before screening for "likely
voters" and regardless of the response rate,
those willing to complete an IVR study may well represent a population that is
better informed or more politically interested than those who complete a survey
with an interviewer.
Put this all together, and it is clear that the Rasmussen
survey is reaching a very different population, something I would wager
explains much of the difference in the results charted above.
Now, the second question, which result is more "accurate?" It is tempting to say that this question is impossible to
answer, since we will never have a national primary election to check it against.
But a better answer may be that "accuracy" in this case depends on what we want
to use the data for.
If we were trying to predict the outcome of a national
primary, and if all other aspects of methodology were equal (which they're
not), I would want to look at the narrower slice of "likely voters" rather than
all adult "leaned Republicans." Since the nomination process involves series of
primaries and caucuses starting with Iowa and New Hampshire, and since
the results from those early contests typically influence preferences in the
states that vote later, we really need to focus on early states for a more
"accurate" assessment of where things stand now. While interesting and fun to
follow, these national measurements provide only indirect indicators of the
current status of the race for the White House.
Why would the ABC/Post survey want to look at all
Republicans, rather than likely voters? Here is the way ABC polling director
Gary Langer explained it in his online
column this week:
I like to think there are two things we cover in
an election campaign. One is the election; the other is the campaign.The campaign is about who wins. It's about tactics
and strategy, fundraising and ad buys, endorsements and get-out-the-vote
drives. It's about the score of the game - the horse race, contest-by-contest,
and nothing else. We cover it, as we should.The election is the bigger picture: It's about
Americans coming together in their quadrennial exercise of democracy - sizing
up where we're at as a country, where we want to be and what kind of person
we'd like to lead us there. It's a different story than the horse race, with
more texture to it, and plenty of meaning. We cover it, too.We ask the horse race question in our national
polls for context - not to predict the winner of a made-up national primary,
but to see how views on issues, candidate attributes and the public's personal
characteristics inform their preferences.
Questions like Andrew's are more consequential in the statewide surveys we
are tracking here at Pollster.com, and those surveys have been producing some
discrepancies even bigger than the one charted above. We will all be in a
better to make sense of those differences if we know more about the
methodologies pollsters use. I'll be turning to that issue in far more detail
next week.