Two new polls released yesterday asked about a hypothetical presidential contest between Sarah Palin and Barack Obama with very different results: The Time/SRBI poll (article, SRBI analysis & results) shows Obama with a massive, 21-point advantage (55% to 34% with 11% unsure or not voting), while a survey by Public Policy Polling (PPP) shows a dead heat (46% to 46% with 9% undecided).
What gives? And which poll, if any, should we trust?
The two surveys were conducted by telephone just a few days apart (July 9-12 for PPP and July 12-13 for Time/SRBI) and sought to sample registered voters nationwide, but beyond those characteristics, they were very different surveys.
Time/SRBI uses live interviewers. PPP uses an automated, recorded-voice method that asks respondents to answer questions by pressing buttons on their touch-tone phones.
Their sampling methods are also very different: Time/SRBI used a method that selects telephone area codes and exchanges and randomizes the final digits of each number to theoretically reach a random sample of all working telephone numbers. In this case, they drew two samples, one of landline phones and one of cell phones, dialed each separately and combined the two with weighting. They attempted to select a random person in each household and ultimately questioned 1,003 adults (of whom 50 were interviewed by cellphone), although they asked the presidential vote question only of the 87% who said they are registered to vote.
PPP draws random samples of households from a list of all registered voters compiled by Aristotle International using the public lists gathered by voter registrars nationwide. Phone numbers are either provided by voters when they register or obtained by matching addresses to published phone directories, so some undisclosed percentage of the sampled voters lacks a phone number. PPP then interviews whomever answers the phone and asks respondents to “please hang up” if they are not registered to vote. They ultimately interviewed 667 registered voters.
The trade-off: Time/SRBI theoretically covers every registered voter, while PPP misses some undisclosed percentage of voters without phone numbers or that live in cell-phone-only households (federal regulations prohibit pollsters from using an “autodialier” to dial cell phone numbers). On the other hand, PPP’s identification of truly registered voters may be more accurate; self-reports tend to exaggerate the number of registered voters.
The PPP survey featured 17 questions, including demographics. The Time/SRBI survey asked 29 questions, not including demographics. Did the difference in length and mode (interviewer or no interviewer), produce a different response rate? Neither organization released a response rate, so we do not know.
Do all of these characteristics add up to different kinds of people interviewed? One big clue comes from the results for party identification: PPP’s respondents identified themselves as 39% Democrat, 34% Republican and 27% independent or other. Time/SRBI provided me with their party ID results for registered voters: On the initial question, 33% say they “usually think of” themselves as Democrats, 23% as Republicans, 30% as independents, 12% as “something else” and 2% were not sure. When they pushed the uncertain, a total of 47% identify or “lean” Democratic and 30% identify or lean Republican.
So the PPP sample has a closer partisan balance than the Time/SRBI sample, although we should keep in mind that the two surveys also asked slightly different party identification questions.
But wait, there’s more: The vote question also differs in an important way. Time/SRBI identifies the party of each candidate, while PPP omits party labels:
Time/SRBI: If the presidential election were held today and the candidates were Barack Obama, the Democrat, and Sarah Palin, the Republican, and you had to choose, for whom would you vote?
PPP: If the candidates for President next time were Sarah Palin and Barack Obama, who would you vote for?
Finally, the two polls asked their Obama-vs-Palin question in slightly different contexts. Time/SBRI asked their question following a set of probes of Obama’s performance as president, a question about whether Obama or George W. Bush “is the better president” and immediately following a job rating of first lady Michelle Obama. PPP asked their Obama-Palin question after a job rating of Obama, favorable ratings of each of the Republicans and immediately after a Huckabee-vs-Obama question.
The main point here is that these polls are very different in ways that go far beyond live interviewers versus automated polling. Their methods are dissimilar across the board.
So given these differences, which poll should we trust? My answer is neither. Or both.
First the case for both: When an attitude or preference is weak, small differences in methodology and question wording can make a big difference. Needless to say, asking for preferences in a hypothetical political match-up in 2012 two years before an election qualifies as weak. In such cases, it makes sense to look at a wide variety of polls in order to get a sense of the range of potential results.
The case for “neither” is a lot stronger in this instance, for the same reasons. Seven years ago while addressing the American Association for Public Opinion Research (AAPOR), my new boss Arianna Huffington offered a quip about a similar presidential vote preference question asked nearly four years before the 2004 election:
This is really about as meaningful as phrasing a question in the following way, which I will suggest you try one day, “If the world were to stop spinning, and all life were placed in a state of suspended animation, who would you like to see in the Oval Office when you thawed out?”
Yes, I’m guilty of sucking up a bit with that reference, but she’s right. How many ordinary voters have thought deeply about a contest between Obama and Palin? How many were forming an opinion on the spot when interviewed, only after hearing the question posed over the telephone?
My best advice to anyone trying to understand Sarah Palin’s potential is to put aside these two measures and focus instead on questions about opinions that are closer to real, such as Palin’s favorability rating (as asked in both the PPP survey and the results released by Gallup earlier today). Ordinary people do have genuine, pre-existing opinions about Barack Obama and Sarah Palin. Polls are on most solid ground when they measure these perceptions separately, rather than asking about a still hypothetical contest that has so far been of interest mostly to political junkies.